2023-07-14T17:58:47.029 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T17:58:47.030 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010 -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/orig.config.yaml 2023-07-14T17:58:47.088 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T17:59:02.198 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7338010' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi064.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi144.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup random 0.01 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T17:59:02.199 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T17:59:02.200 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T17:59:02.201 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T17:59:02.204 INFO:teuthology.task.internal:Checking packages... 2023-07-14T17:59:02.242 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T17:59:02.242 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T17:59:02.243 INFO:teuthology.packaging:ref: None 2023-07-14T17:59:02.244 INFO:teuthology.packaging:tag: None 2023-07-14T17:59:02.244 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T17:59:02.245 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T17:59:02.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T17:59:03.357 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T17:59:03.358 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T17:59:03.365 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T17:59:03.365 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T17:59:03.445 INFO:teuthology.task.internal:Saving configuration 2023-07-14T17:59:03.465 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T17:59:03.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T17:59:08.556 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.174558', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:59:11.777 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.170767', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:59:11.799 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.173020', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T17:59:11.800 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T17:59:11.807 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T17:59:11.807 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T17:59:11.807 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T17:59:11.808 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T17:59:11.916 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8aea5ce430>, signals=[15]) 2023-07-14T17:59:11.917 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T17:59:11.924 INFO:teuthology.task.internal:Opening connections... 2023-07-14T17:59:11.925 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T17:59:11.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:59:12.003 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2023-07-14T17:59:12.004 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:59:12.072 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2023-07-14T17:59:12.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T17:59:12.144 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T17:59:12.154 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2023-07-14T17:59:12.175 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2023-07-14T17:59:12.175 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:ID="rhel" 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="fedora" 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="8.6" 2023-07-14T17:59:12.235 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:59:12.236 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:59:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T17:59:12.237 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:59:12.237 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:59:12.237 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:59:12.237 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:59:12.238 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2023-07-14T17:59:12.265 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2023-07-14T17:59:12.284 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2023-07-14T17:59:12.284 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2023-07-14T17:59:12.342 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:59:12.342 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:59:12.342 INFO:teuthology.orchestra.run.smithi105.stdout:ID="rhel" 2023-07-14T17:59:12.342 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="fedora" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="8.6" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:59:12.343 INFO:teuthology.orchestra.run.smithi105.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:59:12.344 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:59:12.345 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2023-07-14T17:59:12.365 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2023-07-14T17:59:12.385 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2023-07-14T17:59:12.385 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:ID="rhel" 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="fedora" 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="8.6" 2023-07-14T17:59:12.443 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el8" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T17:59:12.444 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T17:59:12.445 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T17:59:12.445 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T17:59:12.445 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T17:59:12.445 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T17:59:12.445 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2023-07-14T17:59:12.465 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T17:59:12.476 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T17:59:12.485 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T17:59:12.486 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:59:12.489 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:59:12.491 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2023-07-14T17:59:12.508 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T17:59:12.516 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T17:59:12.517 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:59:12.549 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:59:12.551 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2023-07-14T17:59:12.577 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T17:59:12.666 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T17:59:12.682 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-14T17:59:12.683 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-14T17:59:12.684 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:59:12.685 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:59:12.685 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T17:59:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:12.703 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2023-07-14T17:59:12.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:12.704 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2023-07-14T17:59:12.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:12.705 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2023-07-14T17:59:12.761 INFO:teuthology.orchestra.run.smithi144.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.761 INFO:teuthology.task.kernel:Running kernel on smithi144: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.761 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum install -y kernel 2023-07-14T17:59:12.762 INFO:teuthology.orchestra.run.smithi105.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.763 INFO:teuthology.task.kernel:Running kernel on smithi105: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.763 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2023-07-14T17:59:12.764 INFO:teuthology.orchestra.run.smithi064.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.765 INFO:teuthology.task.kernel:Running kernel on smithi064: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:12.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum install -y kernel 2023-07-14T17:59:13.903 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:13.903 INFO:teuthology.orchestra.run.smithi144.stdout:Unable to read consumer identity 2023-07-14T17:59:13.966 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:13.966 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-07-14T17:59:13.987 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:13.988 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2023-07-14T17:59:15.572 INFO:teuthology.orchestra.run.smithi064.stdout:Copr repo for python3-asyncssh owned by ceph 9.7 kB/s | 3.5 kB 00:00 2023-07-14T17:59:15.574 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 10 kB/s | 3.5 kB 00:00 2023-07-14T17:59:15.575 INFO:teuthology.orchestra.run.smithi144.stdout:Copr repo for python3-asyncssh owned by ceph 9.3 kB/s | 3.5 kB 00:00 2023-07-14T17:59:19.398 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2023-07-14T17:59:21.458 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 2.7 MB/s | 16 MB 00:05 2023-07-14T17:59:22.675 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 2.2 MB/s | 16 MB 00:07 2023-07-14T17:59:23.048 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-07-14T17:59:23.767 INFO:teuthology.orchestra.run.smithi064.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:59:23.819 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-07-14T17:59:23.820 INFO:teuthology.orchestra.run.smithi064.stdout:Nothing to do. 2023-07-14T17:59:23.821 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-07-14T17:59:23.962 DEBUG:teuthology.orchestra.run.smithi064:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:59:24.314 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:24.315 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-07-14T17:59:24.594 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:59:22 PM UTC. 2023-07-14T17:59:24.640 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:24.688 INFO:teuthology.orchestra.run.smithi064.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:24.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum reinstall -y kernel || true 2023-07-14T17:59:24.909 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 414 kB/s | 24 kB 00:00 2023-07-14T17:59:25.084 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:25.085 INFO:teuthology.orchestra.run.smithi064.stdout:Unable to read consumer identity 2023-07-14T17:59:25.359 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 05:59:22 PM UTC. 2023-07-14T17:59:25.401 INFO:teuthology.orchestra.run.smithi064.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:25.428 INFO:teuthology.orchestra.run.smithi064.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:25.480 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:59:25.562 INFO:teuthology.orchestra.run.smithi064.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:25.563 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:25.563 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:59:25.563 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-14T17:59:25.563 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:59:25.611 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:59:25.617 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2023-07-14T17:59:25.630 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:25.659 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-07-14T17:59:25.660 INFO:teuthology.orchestra.run.smithi144.stdout:Nothing to do. 2023-07-14T17:59:25.661 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-07-14T17:59:25.792 DEBUG:teuthology.orchestra.run.smithi144:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:59:26.123 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:26.124 INFO:teuthology.orchestra.run.smithi144.stdout:Unable to read consumer identity 2023-07-14T17:59:26.125 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-07-14T17:59:26.392 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:59:24 PM UTC. 2023-07-14T17:59:26.433 INFO:teuthology.orchestra.run.smithi144.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:26.472 INFO:teuthology.orchestra.run.smithi144.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:26.519 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum reinstall -y kernel || true 2023-07-14T17:59:26.761 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T17:59:26.804 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-07-14T17:59:26.805 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2023-07-14T17:59:26.805 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-07-14T17:59:26.847 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:26.847 INFO:teuthology.orchestra.run.smithi144.stdout:Unable to read consumer identity 2023-07-14T17:59:26.930 DEBUG:teuthology.orchestra.run.smithi105:> echo no | sudo yum reinstall kernel || true 2023-07-14T17:59:27.112 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 05:59:24 PM UTC. 2023-07-14T17:59:27.152 INFO:teuthology.orchestra.run.smithi144.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:27.189 INFO:teuthology.orchestra.run.smithi144.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:27.235 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:59:27.301 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:27.301 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2023-07-14T17:59:27.315 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:27.316 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:27.316 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:59:27.316 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-07-14T17:59:27.317 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:59:27.352 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2023-07-14T17:59:27.368 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:27.567 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 05:59:26 PM UTC. 2023-07-14T17:59:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:27.641 INFO:teuthology.orchestra.run.smithi105.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:27.686 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum reinstall -y kernel || true 2023-07-14T17:59:28.012 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T17:59:28.013 INFO:teuthology.orchestra.run.smithi105.stdout:Unable to read consumer identity 2023-07-14T17:59:28.288 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 05:59:26 PM UTC. 2023-07-14T17:59:28.330 INFO:teuthology.orchestra.run.smithi105.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T17:59:28.353 INFO:teuthology.orchestra.run.smithi105.stderr:Error: No packages marked for reinstall. 2023-07-14T17:59:28.404 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T17:59:28.483 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:28.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T17:59:28.484 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T17:59:28.484 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-14T17:59:28.484 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T17:59:28.515 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2023-07-14T17:59:28.531 DEBUG:teuthology.parallel:result is None 2023-07-14T17:59:28.531 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T17:59:28.541 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T17:59:28.542 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:59:28.544 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:59:28.573 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T17:59:28.593 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T17:59:28.599 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T17:59:28.640 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T17:59:28.640 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:59:28.643 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:59:28.644 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T17:59:28.672 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T17:59:28.679 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T17:59:28.679 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:59:28.704 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:59:28.706 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T17:59:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.742 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.742 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.748 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.753 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.755 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T17:59:28.757 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T17:59:28.763 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T17:59:28.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:59:28.799 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:59:28.801 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T17:59:28.832 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T17:59:28.841 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T17:59:28.841 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:59:28.870 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:59:28.872 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T17:59:28.889 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:33.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:34.019 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.106 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.150 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T17:59:34.151 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:59:34.221 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:34.272 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:34.325 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.418 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.453 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T17:59:34.454 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:59:34.521 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:34.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T17:59:34.627 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T17:59:34.760 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T17:59:34.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T17:59:34.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2023-07-14T17:59:34.833 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2023-07-14T17:59:34.836 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2023-07-14T17:59:34.892 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:59:34.893 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:59:34.929 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T17:59:35.320 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T17:59:35.328 INFO:teuthology.task.internal:Starting timer... 2023-07-14T17:59:35.329 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T17:59:35.359 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T17:59:35.369 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T17:59:35.370 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2023-07-14T17:59:35.430 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2023-07-14T17:59:35.432 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2023-07-14T17:59:35.494 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2023-07-14T17:59:35.496 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2023-07-14T17:59:35.562 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2023-07-14T17:59:35.565 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T17:59:35.565 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2023-07-14T17:59:35.588 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2023-07-14T17:59:35.589 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2023-07-14T17:59:35.616 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2023-07-14T17:59:35.617 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2023-07-14T17:59:35.644 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2023-07-14T17:59:35.645 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2023-07-14T17:59:35.645 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:59:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:35.674 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:59:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:35.700 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T17:59:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T17:59:35.728 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T17:59:35.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:59:35.757 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:59:35.783 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2023-07-14T17:59:35.810 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T17:59:35.819 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-14T17:59:35.825 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-14T17:59:35.971 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T17:59:35.995 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T17:59:35.996 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi064.front.sepia.ceph.com,smithi105.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T18:10:07.101 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2023-07-14T18:10:07.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-07-14T18:10:07.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:10:07.178 DEBUG:teuthology.orchestra.run.smithi064:> true 2023-07-14T18:10:07.245 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2023-07-14T18:10:07.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-07-14T18:10:07.246 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:10:07.315 DEBUG:teuthology.orchestra.run.smithi105:> true 2023-07-14T18:10:07.382 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2023-07-14T18:10:07.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-07-14T18:10:07.383 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:10:07.451 DEBUG:teuthology.orchestra.run.smithi144:> true 2023-07-14T18:10:07.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2023-07-14T18:10:07.520 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T18:10:07.533 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T18:10:07.534 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:10:07.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:10:07.538 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:10:07.539 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:10:07.541 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:10:07.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:10:07.569 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:10:07.572 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:10:07.585 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:10:07.587 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:10:07.592 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:10:07.607 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:10:07.650 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2023-07-14T18:10:07.658 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2023-07-14T18:10:07.661 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2023-07-14T18:10:07.670 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2023-07-14T18:10:07.676 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:10:07.685 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:10:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:10:07.693 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2023-07-14T18:10:07.700 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:10:07.705 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2023-07-14T18:10:07.719 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:10:07.733 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:10:07.768 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2023-07-14T18:10:07.771 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:10:07.771 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2023-07-14T18:10:07.771 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.772 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.772 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.772 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.773 INFO:teuthology.orchestra.run.smithi064.stderr:bash: ntpq: command not found 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.776 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.809 INFO:teuthology.orchestra.run.smithi144.stderr:bash: ntpq: command not found 2023-07-14T18:10:07.813 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:10:07.813 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-07-14T18:10:07.814 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.814 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.814 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.814 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:10:07.815 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T18:10:07.824 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T18:10:07.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:10:07.825 DEBUG:teuthology.orchestra.run.smithi064:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:10:07.825 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi105.front.sepia.ceph.com 2023-07-14T18:10:07.826 DEBUG:teuthology.orchestra.run.smithi105:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:10:07.826 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi144.front.sepia.ceph.com 2023-07-14T18:10:07.826 DEBUG:teuthology.orchestra.run.smithi144:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:10:08.196 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:08.226 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:09.070 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:29 ago on Fri 14 Jul 2023 06:07:39 PM UTC. 2023-07-14T18:10:09.120 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:31 ago on Fri 14 Jul 2023 06:07:37 PM UTC. 2023-07-14T18:10:09.141 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:47 ago on Fri 14 Jul 2023 06:08:21 PM UTC. 2023-07-14T18:10:10.785 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-07-14T18:10:10.786 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout:Resetting modules: 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout: container-tools 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:10.787 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-07-14T18:10:10.788 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2023-07-14T18:10:10.788 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:10.856 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout:Resetting modules: 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout: container-tools 2023-07-14T18:10:10.857 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:10.858 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-07-14T18:10:10.858 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2023-07-14T18:10:10.858 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:11.099 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-07-14T18:10:11.100 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: Package Architecture Version Repository Size 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout:Resetting modules: 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: container-tools 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2023-07-14T18:10:11.101 INFO:teuthology.orchestra.run.smithi064.stdout:================================================================================ 2023-07-14T18:10:11.102 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:11.759 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:32 ago on Fri 14 Jul 2023 06:07:39 PM UTC. 2023-07-14T18:10:11.833 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:34 ago on Fri 14 Jul 2023 06:07:37 PM UTC. 2023-07-14T18:10:12.090 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:50 ago on Fri 14 Jul 2023 06:08:21 PM UTC. 2023-07-14T18:10:14.454 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-07-14T18:10:14.620 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-07-14T18:10:14.864 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-07-14T18:10:14.899 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:15.070 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:15.347 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:36 ago on Fri 14 Jul 2023 06:07:39 PM UTC. 2023-07-14T18:10:15.848 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:38 ago on Fri 14 Jul 2023 06:07:37 PM UTC. 2023-07-14T18:10:16.187 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:01:54 ago on Fri 14 Jul 2023 06:08:21 PM UTC. 2023-07-14T18:10:17.369 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-07-14T18:10:17.461 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================================ 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================================ 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout:Installing group/module packages: 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:10:17.462 INFO:teuthology.orchestra.run.smithi105.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:10:17.463 INFO:teuthology.orchestra.run.smithi105.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout:Installing module profiles: 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout: container-tools/common 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling module streams: 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout: container-tools rhel8 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-07-14T18:10:17.464 INFO:teuthology.orchestra.run.smithi105.stdout:============================================================================================================ 2023-07-14T18:10:17.465 INFO:teuthology.orchestra.run.smithi105.stdout:Install 10 Packages 2023-07-14T18:10:17.465 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:17.465 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 18 M 2023-07-14T18:10:17.465 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 65 M 2023-07-14T18:10:17.466 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-07-14T18:10:17.546 INFO:teuthology.orchestra.run.smithi105.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 304 kB/s | 25 kB 00:00 2023-07-14T18:10:17.562 INFO:teuthology.orchestra.run.smithi105.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 952 kB/s | 94 kB 00:00 2023-07-14T18:10:17.588 INFO:teuthology.orchestra.run.smithi105.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 330 kB/s | 41 kB 00:00 2023-07-14T18:10:17.588 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-07-14T18:10:17.613 INFO:teuthology.orchestra.run.smithi105.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 736 kB/s | 49 kB 00:00 2023-07-14T18:10:17.647 INFO:teuthology.orchestra.run.smithi105.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-14T18:10:17.672 INFO:teuthology.orchestra.run.smithi105.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================================ 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================================ 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout:Installing group/module packages: 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:10:17.683 INFO:teuthology.orchestra.run.smithi144.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2023-07-14T18:10:17.684 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout:Installing module profiles: 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout: container-tools/common 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout:Enabling module streams: 2023-07-14T18:10:17.685 INFO:teuthology.orchestra.run.smithi144.stdout: container-tools rhel8 2023-07-14T18:10:17.686 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:17.686 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-07-14T18:10:17.686 INFO:teuthology.orchestra.run.smithi144.stdout:============================================================================================================ 2023-07-14T18:10:17.686 INFO:teuthology.orchestra.run.smithi144.stdout:Install 10 Packages 2023-07-14T18:10:17.686 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:17.687 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 18 M 2023-07-14T18:10:17.687 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 65 M 2023-07-14T18:10:17.687 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-07-14T18:10:17.697 INFO:teuthology.orchestra.run.smithi105.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-14T18:10:17.760 INFO:teuthology.orchestra.run.smithi144.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 549 kB/s | 41 kB 00:00 2023-07-14T18:10:17.785 INFO:teuthology.orchestra.run.smithi144.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 249 kB/s | 25 kB 00:00 2023-07-14T18:10:17.806 INFO:teuthology.orchestra.run.smithi105.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 20 MB/s | 2.2 MB 00:00 2023-07-14T18:10:17.810 INFO:teuthology.orchestra.run.smithi144.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 983 kB/s | 49 kB 00:00 2023-07-14T18:10:17.836 INFO:teuthology.orchestra.run.smithi144.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 624 kB/s | 94 kB 00:00 2023-07-14T18:10:17.869 INFO:teuthology.orchestra.run.smithi144.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-07-14T18:10:17.914 INFO:teuthology.orchestra.run.smithi105.stdout:(9/10): buildah-1.26.2-1.module+el8.6.0+16771+2 24 MB/s | 8.0 MB 00:00 2023-07-14T18:10:17.918 INFO:teuthology.orchestra.run.smithi144.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.8 MB/s | 149 kB 00:00 2023-07-14T18:10:17.953 INFO:teuthology.orchestra.run.smithi144.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 2.5 MB/s | 213 kB 00:00 2023-07-14T18:10:18.006 INFO:teuthology.orchestra.run.smithi105.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 21 MB/s | 7.0 MB 00:00 2023-07-14T18:10:18.007 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:10:18.007 INFO:teuthology.orchestra.run.smithi105.stdout:Total 34 MB/s | 18 MB 00:00 2023-07-14T18:10:18.045 INFO:teuthology.orchestra.run.smithi144.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-14T18:10:18.071 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-07-14T18:10:18.171 INFO:teuthology.orchestra.run.smithi144.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 28 MB/s | 7.0 MB 00:00 2023-07-14T18:10:18.172 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================================ 2023-07-14T18:10:18.172 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2023-07-14T18:10:18.172 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================================ 2023-07-14T18:10:18.172 INFO:teuthology.orchestra.run.smithi064.stdout:Installing group/module packages: 2023-07-14T18:10:18.173 INFO:teuthology.orchestra.run.smithi064.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:10:18.173 INFO:teuthology.orchestra.run.smithi064.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:10:18.173 INFO:teuthology.orchestra.run.smithi064.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:10:18.174 INFO:teuthology.orchestra.run.smithi064.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:10:18.174 INFO:teuthology.orchestra.run.smithi064.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:10:18.174 INFO:teuthology.orchestra.run.smithi064.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:10:18.174 INFO:teuthology.orchestra.run.smithi064.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:10:18.175 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2023-07-14T18:10:18.175 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:10:18.175 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:10:18.175 INFO:teuthology.orchestra.run.smithi064.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:10:18.176 INFO:teuthology.orchestra.run.smithi064.stdout:Installing module profiles: 2023-07-14T18:10:18.176 INFO:teuthology.orchestra.run.smithi064.stdout: container-tools/common 2023-07-14T18:10:18.176 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:18.176 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling module streams: 2023-07-14T18:10:18.177 INFO:teuthology.orchestra.run.smithi064.stdout: container-tools rhel8 2023-07-14T18:10:18.177 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:18.178 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2023-07-14T18:10:18.178 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================================ 2023-07-14T18:10:18.178 INFO:teuthology.orchestra.run.smithi064.stdout:Install 10 Packages 2023-07-14T18:10:18.178 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:18.179 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 18 M 2023-07-14T18:10:18.179 INFO:teuthology.orchestra.run.smithi064.stdout:Installed size: 65 M 2023-07-14T18:10:18.180 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2023-07-14T18:10:18.205 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-07-14T18:10:18.224 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-07-14T18:10:18.224 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-07-14T18:10:18.245 INFO:teuthology.orchestra.run.smithi064.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.3 MB/s | 94 kB 00:00 2023-07-14T18:10:18.262 INFO:teuthology.orchestra.run.smithi144.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 18 MB/s | 8.0 MB 00:00 2023-07-14T18:10:18.263 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:10:18.263 INFO:teuthology.orchestra.run.smithi144.stdout:Total 32 MB/s | 18 MB 00:00 2023-07-14T18:10:18.271 INFO:teuthology.orchestra.run.smithi064.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 257 kB/s | 25 kB 00:00 2023-07-14T18:10:18.296 INFO:teuthology.orchestra.run.smithi064.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 335 kB/s | 41 kB 00:00 2023-07-14T18:10:18.321 INFO:teuthology.orchestra.run.smithi064.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 653 kB/s | 49 kB 00:00 2023-07-14T18:10:18.380 INFO:teuthology.orchestra.run.smithi064.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 4.7 MB/s | 524 kB 00:00 2023-07-14T18:10:18.397 INFO:teuthology.orchestra.run.smithi064.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.9 MB/s | 149 kB 00:00 2023-07-14T18:10:18.405 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-07-14T18:10:18.409 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-07-14T18:10:18.430 INFO:teuthology.orchestra.run.smithi064.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-07-14T18:10:18.464 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-07-14T18:10:18.483 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-07-14T18:10:18.483 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-07-14T18:10:18.523 INFO:teuthology.orchestra.run.smithi064.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-07-14T18:10:18.664 INFO:teuthology.orchestra.run.smithi064.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 26 MB/s | 7.0 MB 00:00 2023-07-14T18:10:18.665 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-07-14T18:10:18.668 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-07-14T18:10:18.756 INFO:teuthology.orchestra.run.smithi064.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 17 MB/s | 8.0 MB 00:00 2023-07-14T18:10:18.757 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:10:18.757 INFO:teuthology.orchestra.run.smithi064.stdout:Total 31 MB/s | 18 MB 00:00 2023-07-14T18:10:18.893 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-07-14T18:10:18.962 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2023-07-14T18:10:18.983 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2023-07-14T18:10:18.983 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2023-07-14T18:10:19.076 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:10:19.176 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2023-07-14T18:10:19.180 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2023-07-14T18:10:19.188 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-07-14T18:10:19.248 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:19.363 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:10:19.425 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:10:19.602 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:19.644 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:10:19.713 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2023-07-14T18:10:19.796 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:10:19.904 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:10:19.957 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:10:20.009 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:10:20.062 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:10:20.322 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:10:20.499 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:10:20.779 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:10:20.868 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:10:21.142 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:10:21.652 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:10:21.659 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:10:21.869 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:10:22.010 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:10:22.054 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:10:22.239 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:10:22.273 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:22.401 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:10:22.541 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:10:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:10:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:10:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:10:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:10:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:10:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:10:22.543 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:10:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:10:22.578 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:22.779 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:10:22.848 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:22.848 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:10:22.849 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:22.849 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:10:22.849 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:10:22.850 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:10:22.850 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:10:22.850 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:10:22.851 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:10:22.851 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:10:22.968 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:10:23.152 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:10:23.437 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:10:23.438 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:10:23.438 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:10:23.438 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:10:23.438 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:10:23.653 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:10:23.653 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:44 ago on Fri 14 Jul 2023 06:07:39 PM UTC. 2023-07-14T18:10:23.965 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:10:23.965 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:46 ago on Fri 14 Jul 2023 06:07:37 PM UTC. 2023-07-14T18:10:24.618 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:10:24.618 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:02:03 ago on Fri 14 Jul 2023 06:08:21 PM UTC. 2023-07-14T18:10:27.671 INFO:teuthology.orchestra.run.smithi144.stdout:Installed products updated. 2023-07-14T18:10:27.867 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:27.867 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2023-07-14T18:10:27.867 INFO:teuthology.orchestra.run.smithi144.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:27.867 INFO:teuthology.orchestra.run.smithi144.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:27.867 INFO:teuthology.orchestra.run.smithi144.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:27.868 INFO:teuthology.orchestra.run.smithi144.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:10:27.869 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:10:27.869 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-07-14T18:10:28.008 DEBUG:teuthology.parallel:result is None 2023-07-14T18:10:28.208 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2023-07-14T18:10:28.397 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:10:28.398 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:10:28.399 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-07-14T18:10:28.536 DEBUG:teuthology.parallel:result is None 2023-07-14T18:10:29.305 INFO:teuthology.orchestra.run.smithi064.stdout:Installed products updated. 2023-07-14T18:10:29.519 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:29.519 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2023-07-14T18:10:29.519 INFO:teuthology.orchestra.run.smithi064.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:29.519 INFO:teuthology.orchestra.run.smithi064.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:10:29.520 INFO:teuthology.orchestra.run.smithi064.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:10:29.521 INFO:teuthology.orchestra.run.smithi064.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:10:29.521 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:10:29.521 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-07-14T18:10:29.884 DEBUG:teuthology.parallel:result is None 2023-07-14T18:10:29.884 INFO:teuthology.run_tasks:Running task install... 2023-07-14T18:10:29.893 DEBUG:teuthology.task.install:project ceph 2023-07-14T18:10:29.894 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T18:10:29.894 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T18:10:29.894 INFO:teuthology.task.install:Using flavor: default 2023-07-14T18:10:29.900 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T18:10:29.901 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T18:10:29.901 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:10:29.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:10:29.904 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:10:29.905 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:10:29.906 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:10:29.906 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:10:40.277 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:10:40.277 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:10:45.287 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:10:45.287 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:10:45.289 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:10:45.289 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:10:55.653 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:10:55.654 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:10:55.654 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:10:55.656 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:10:55.656 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:10:55.657 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:10:55.690 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:10:55.690 DEBUG:teuthology.orchestra.run.smithi144:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:10:55.692 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:10:55.692 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:10:55.725 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:10:55.726 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:10:55.726 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:10:55.760 DEBUG:teuthology.orchestra.run.smithi144:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:10:55.762 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:10:55.762 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:10:55.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:10:55.833 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:10:55.840 DEBUG:teuthology.orchestra.run.smithi144:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:10:55.849 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:10:55.878 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2023-07-14T18:10:55.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2023-07-14T18:10:55.907 INFO:teuthology.orchestra.run.smithi144.stdout:check_obsoletes = 1 2023-07-14T18:10:55.908 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum clean all 2023-07-14T18:10:55.912 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:10:55.982 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2023-07-14T18:10:55.983 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2023-07-14T18:10:56.274 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:56.278 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:56.361 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:57.225 INFO:teuthology.orchestra.run.smithi064.stdout:44 files removed 2023-07-14T18:10:57.237 INFO:teuthology.orchestra.run.smithi144.stdout:44 files removed 2023-07-14T18:10:57.278 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:10:57.290 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:10:57.494 INFO:teuthology.orchestra.run.smithi105.stdout:44 files removed 2023-07-14T18:10:57.545 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:10:57.623 INFO:teuthology.orchestra.run.smithi144.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:57.630 INFO:teuthology.orchestra.run.smithi064.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:57.875 INFO:teuthology.orchestra.run.smithi105.stdout:Updating Subscription Management repositories. 2023-07-14T18:10:59.257 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 107 kB/s | 81 kB 00:00 2023-07-14T18:10:59.322 INFO:teuthology.orchestra.run.smithi144.stdout:ceph packages for x86_64 80 kB/s | 81 kB 00:01 2023-07-14T18:10:59.467 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 71 kB/s | 81 kB 00:01 2023-07-14T18:10:59.941 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-14T18:11:00.006 INFO:teuthology.orchestra.run.smithi144.stdout:ceph noarch packages 14 kB/s | 9.4 kB 00:00 2023-07-14T18:11:00.101 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 16 kB/s | 9.4 kB 00:00 2023-07-14T18:11:00.665 INFO:teuthology.orchestra.run.smithi144.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2023-07-14T18:11:00.743 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 2.4 kB/s | 1.5 kB 00:00 2023-07-14T18:11:00.915 INFO:teuthology.orchestra.run.smithi144.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-07-14T18:11:00.991 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 1.5 kB/s | 1.5 kB 00:01 2023-07-14T18:11:01.052 INFO:teuthology.orchestra.run.smithi064.stdout:Copr repo for python3-asyncssh owned by ceph 12 kB/s | 3.5 kB 00:00 2023-07-14T18:11:01.284 INFO:teuthology.orchestra.run.smithi105.stdout:Copr repo for python3-asyncssh owned by ceph 13 kB/s | 3.5 kB 00:00 2023-07-14T18:11:02.579 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 16 MB 00:01 2023-07-14T18:11:03.795 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 16 MB 00:02 2023-07-14T18:11:05.983 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 369 kB/s | 24 kB 00:00 2023-07-14T18:11:06.902 INFO:teuthology.orchestra.run.smithi144.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 47 MB 00:00 2023-07-14T18:11:07.218 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 478 kB/s | 24 kB 00:00 2023-07-14T18:11:08.137 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-07-14T18:11:14.819 INFO:teuthology.orchestra.run.smithi144.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-07-14T18:11:16.204 INFO:teuthology.orchestra.run.smithi105.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 56 MB/s | 47 MB 00:00 2023-07-14T18:11:24.261 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 689 kB/s | 16 MB 00:23 2023-07-14T18:11:25.950 INFO:teuthology.orchestra.run.smithi144.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:25.950 INFO:teuthology.orchestra.run.smithi144.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:25.966 INFO:teuthology.orchestra.run.smithi144.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:25.967 INFO:teuthology.orchestra.run.smithi144.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:25.968 INFO:teuthology.orchestra.run.smithi144.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:11:25.969 INFO:teuthology.orchestra.run.smithi144.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:11:25.971 INFO:teuthology.orchestra.run.smithi144.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:11:25.975 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:11:25.976 INFO:teuthology.orchestra.run.smithi144.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:11:26.109 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2023-07-14T18:11:26.121 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================================================== 2023-07-14T18:11:26.121 INFO:teuthology.orchestra.run.smithi144.stdout: Package Arch Version Repository Size 2023-07-14T18:11:26.122 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================================================== 2023-07-14T18:11:26.122 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2023-07-14T18:11:26.122 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:11:26.122 INFO:teuthology.orchestra.run.smithi144.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:11:26.123 INFO:teuthology.orchestra.run.smithi144.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:11:26.123 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:11:26.123 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:11:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:11:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:11:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:11:26.124 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:11:26.125 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:11:26.125 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:11:26.125 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:11:26.126 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:11:26.126 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:11:26.126 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:11:26.126 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:11:26.127 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:11:26.127 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:11:26.127 INFO:teuthology.orchestra.run.smithi144.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:11:26.128 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:11:26.128 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:11:26.128 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:11:26.128 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:11:26.129 INFO:teuthology.orchestra.run.smithi144.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:11:26.129 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:11:26.129 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:11:26.130 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:11:26.130 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:11:26.130 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:11:26.130 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:11:26.131 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:11:26.131 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:11:26.131 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:11:26.132 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:11:26.132 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:11:26.132 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:11:26.133 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:11:26.133 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrading: 2023-07-14T18:11:26.133 INFO:teuthology.orchestra.run.smithi144.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:11:26.133 INFO:teuthology.orchestra.run.smithi144.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:11:26.134 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:11:26.134 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2023-07-14T18:11:26.134 INFO:teuthology.orchestra.run.smithi144.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:11:26.135 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:11:26.135 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:11:26.135 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:11:26.135 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:11:26.136 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:11:26.136 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:11:26.136 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:11:26.137 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:11:26.137 INFO:teuthology.orchestra.run.smithi144.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:11:26.137 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:11:26.137 INFO:teuthology.orchestra.run.smithi144.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:11:26.138 INFO:teuthology.orchestra.run.smithi144.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:11:26.138 INFO:teuthology.orchestra.run.smithi144.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:11:26.138 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:11:26.139 INFO:teuthology.orchestra.run.smithi144.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:11:26.139 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:11:26.139 INFO:teuthology.orchestra.run.smithi144.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:11:26.139 INFO:teuthology.orchestra.run.smithi144.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:11:26.140 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:11:26.140 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:11:26.140 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:11:26.141 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:11:26.141 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:11:26.141 INFO:teuthology.orchestra.run.smithi144.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:11:26.141 INFO:teuthology.orchestra.run.smithi144.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:11:26.142 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:11:26.142 INFO:teuthology.orchestra.run.smithi144.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:11:26.142 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:11:26.143 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:11:26.143 INFO:teuthology.orchestra.run.smithi144.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:11:26.143 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:11:26.144 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:11:26.144 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:11:26.144 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:11:26.144 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:11:26.145 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:11:26.145 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:11:26.145 INFO:teuthology.orchestra.run.smithi144.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:11:26.146 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:11:26.146 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:11:26.146 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:11:26.146 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:11:26.147 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:11:26.148 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:11:26.148 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:11:26.148 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:11:26.148 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:11:26.148 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:11:26.149 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:11:26.150 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:11:26.151 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:11:26.151 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:11:26.151 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:11:26.151 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:11:26.151 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:11:26.152 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:11:26.152 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:26.152 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:11:26.152 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:11:26.152 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:11:26.153 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:11:26.154 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout:Enabling module streams: 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql 10 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout:=========================================================================================================================== 2023-07-14T18:11:26.155 INFO:teuthology.orchestra.run.smithi144.stdout:Install 128 Packages 2023-07-14T18:11:26.156 INFO:teuthology.orchestra.run.smithi144.stdout:Upgrade 3 Packages 2023-07-14T18:11:26.156 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:11:26.156 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 206 M 2023-07-14T18:11:26.157 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2023-07-14T18:11:26.694 INFO:teuthology.orchestra.run.smithi144.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 22 kB/s | 6.4 kB 00:00 2023-07-14T18:11:27.161 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:27.162 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:27.177 INFO:teuthology.orchestra.run.smithi105.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:27.178 INFO:teuthology.orchestra.run.smithi105.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:27.179 INFO:teuthology.orchestra.run.smithi105.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:11:27.180 INFO:teuthology.orchestra.run.smithi105.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:11:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:11:27.186 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:11:27.187 INFO:teuthology.orchestra.run.smithi105.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:11:27.324 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2023-07-14T18:11:27.336 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================================================== 2023-07-14T18:11:27.336 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2023-07-14T18:11:27.336 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================================================== 2023-07-14T18:11:27.336 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2023-07-14T18:11:27.336 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:11:27.337 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:11:27.338 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:11:27.339 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:11:27.339 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:11:27.339 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:11:27.339 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:11:27.340 INFO:teuthology.orchestra.run.smithi105.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:11:27.340 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:11:27.340 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:11:27.341 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:11:27.341 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:11:27.341 INFO:teuthology.orchestra.run.smithi105.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:11:27.342 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:11:27.342 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:11:27.343 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:11:27.343 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:11:27.343 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:11:27.344 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:11:27.344 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:11:27.344 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:11:27.345 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:11:27.345 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:11:27.345 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:11:27.345 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:11:27.346 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:11:27.346 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2023-07-14T18:11:27.346 INFO:teuthology.orchestra.run.smithi105.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:11:27.347 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:11:27.347 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:11:27.347 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2023-07-14T18:11:27.347 INFO:teuthology.orchestra.run.smithi105.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:11:27.348 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:11:27.348 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:11:27.348 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:11:27.348 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:11:27.349 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:11:27.349 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:11:27.349 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:11:27.350 INFO:teuthology.orchestra.run.smithi105.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:11:27.351 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:11:27.352 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:11:27.353 INFO:teuthology.orchestra.run.smithi105.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:11:27.353 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:11:27.353 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:11:27.353 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:11:27.354 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:11:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:11:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:11:27.357 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:11:27.358 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:11:27.359 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:11:27.360 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:11:27.361 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:11:27.362 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling module streams: 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql 10 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================================================== 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout:Install 128 Packages 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 3 Packages 2023-07-14T18:11:27.363 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:11:27.364 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 206 M 2023-07-14T18:11:27.364 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2023-07-14T18:11:27.453 INFO:teuthology.orchestra.run.smithi144.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.1 MB/s | 850 kB 00:00 2023-07-14T18:11:27.611 INFO:teuthology.orchestra.run.smithi144.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.0 MB/s | 167 kB 00:00 2023-07-14T18:11:27.909 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 403 kB/s | 24 kB 00:00 2023-07-14T18:11:28.038 INFO:teuthology.orchestra.run.smithi105.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 27 kB/s | 6.4 kB 00:00 2023-07-14T18:11:28.370 INFO:teuthology.orchestra.run.smithi144.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 3.0 MB/s | 2.3 MB 00:00 2023-07-14T18:11:28.413 INFO:teuthology.orchestra.run.smithi105.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 2.2 MB/s | 850 kB 00:00 2023-07-14T18:11:28.478 INFO:teuthology.orchestra.run.smithi144.stdout:(5/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 2.6 MB/s | 5.4 MB 00:02 2023-07-14T18:11:28.488 INFO:teuthology.orchestra.run.smithi105.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 2.2 MB/s | 167 kB 00:00 2023-07-14T18:11:28.845 INFO:teuthology.orchestra.run.smithi144.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 3.3 MB/s | 1.6 MB 00:00 2023-07-14T18:11:29.189 INFO:teuthology.orchestra.run.smithi105.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 3.3 MB/s | 2.3 MB 00:00 2023-07-14T18:11:29.261 INFO:teuthology.orchestra.run.smithi064.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 42 MB/s | 53 MB 00:01 2023-07-14T18:11:29.271 INFO:teuthology.orchestra.run.smithi144.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 5.8 MB/s | 4.6 MB 00:00 2023-07-14T18:11:29.364 INFO:teuthology.orchestra.run.smithi105.stdout:(5/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 3.4 MB/s | 5.4 MB 00:01 2023-07-14T18:11:29.773 INFO:teuthology.orchestra.run.smithi105.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 2.7 MB/s | 1.6 MB 00:00 2023-07-14T18:11:31.148 INFO:teuthology.orchestra.run.smithi105.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 2.6 MB/s | 4.6 MB 00:01 2023-07-14T18:11:31.521 INFO:teuthology.orchestra.run.smithi144.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 4.3 MB/s | 22 MB 00:05 2023-07-14T18:11:31.604 INFO:teuthology.orchestra.run.smithi144.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 292 kB/s | 24 kB 00:00 2023-07-14T18:11:32.296 INFO:teuthology.orchestra.run.smithi144.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 4.5 MB/s | 14 MB 00:03 2023-07-14T18:11:32.388 INFO:teuthology.orchestra.run.smithi144.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 2.8 MB/s | 260 kB 00:00 2023-07-14T18:11:32.480 INFO:teuthology.orchestra.run.smithi144.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 339 kB/s | 31 kB 00:00 2023-07-14T18:11:32.697 INFO:teuthology.orchestra.run.smithi144.stdout:(13/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 4.6 MB/s | 18 MB 00:03 2023-07-14T18:11:32.731 INFO:teuthology.orchestra.run.smithi144.stdout:(14/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 3.0 MB/s | 772 kB 00:00 2023-07-14T18:11:32.773 INFO:teuthology.orchestra.run.smithi144.stdout:(15/131): libcephsqlite-18.0.0-4940.g81c9387d.e 1.8 MB/s | 135 kB 00:00 2023-07-14T18:11:32.815 INFO:teuthology.orchestra.run.smithi144.stdout:(16/131): librados-devel-18.0.0-4940.g81c9387d. 1.6 MB/s | 139 kB 00:00 2023-07-14T18:11:32.940 INFO:teuthology.orchestra.run.smithi144.stdout:(17/131): libradosstriper1-18.0.0-4940.g81c9387 2.8 MB/s | 473 kB 00:00 2023-07-14T18:11:33.057 INFO:teuthology.orchestra.run.smithi144.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 383 kB/s | 45 kB 00:00 2023-07-14T18:11:33.149 INFO:teuthology.orchestra.run.smithi144.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 1.3 MB/s | 122 kB 00:00 2023-07-14T18:11:33.233 INFO:teuthology.orchestra.run.smithi144.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 2.3 MB/s | 197 kB 00:00 2023-07-14T18:11:33.358 INFO:teuthology.orchestra.run.smithi144.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 2.9 MB/s | 371 kB 00:00 2023-07-14T18:11:33.458 INFO:teuthology.orchestra.run.smithi144.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 3.6 MB/s | 364 kB 00:00 2023-07-14T18:11:33.517 INFO:teuthology.orchestra.run.smithi144.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 1.8 MB/s | 107 kB 00:00 2023-07-14T18:11:33.540 INFO:teuthology.orchestra.run.smithi105.stdout:(8/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 4.7 MB/s | 18 MB 00:03 2023-07-14T18:11:33.584 INFO:teuthology.orchestra.run.smithi144.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 1.4 MB/s | 94 kB 00:00 2023-07-14T18:11:33.607 INFO:teuthology.orchestra.run.smithi105.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 367 kB/s | 24 kB 00:00 2023-07-14T18:11:34.301 INFO:teuthology.orchestra.run.smithi144.stdout:(25/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 4.6 MB/s | 6.8 MB 00:01 2023-07-14T18:11:34.384 INFO:teuthology.orchestra.run.smithi144.stdout:(26/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 4.4 MB/s | 3.5 MB 00:00 2023-07-14T18:11:34.410 INFO:teuthology.orchestra.run.smithi144.stdout:(27/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.6 MB/s | 179 kB 00:00 2023-07-14T18:11:34.477 INFO:teuthology.orchestra.run.smithi144.stdout:(28/131): ceph-grafana-dashboards-18.0.0-4940.g 251 kB/s | 23 kB 00:00 2023-07-14T18:11:34.502 INFO:teuthology.orchestra.run.smithi144.stdout:(29/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 1.4 MB/s | 129 kB 00:00 2023-07-14T18:11:34.968 INFO:teuthology.orchestra.run.smithi144.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 3.5 MB/s | 1.7 MB 00:00 2023-07-14T18:11:35.098 INFO:teuthology.orchestra.run.smithi144.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.8 MB/s | 241 kB 00:00 2023-07-14T18:11:35.307 INFO:teuthology.orchestra.run.smithi105.stdout:(10/131): ceph-common-18.0.0-4940.g81c9387d.el8 3.0 MB/s | 22 MB 00:07 2023-07-14T18:11:35.536 INFO:teuthology.orchestra.run.smithi144.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 108 kB/s | 47 kB 00:00 2023-07-14T18:11:35.636 INFO:teuthology.orchestra.run.smithi144.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 145 kB/s | 15 kB 00:00 2023-07-14T18:11:35.795 INFO:teuthology.orchestra.run.smithi144.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.6 MB/s | 258 kB 00:00 2023-07-14T18:11:36.032 INFO:teuthology.orchestra.run.smithi105.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 359 kB/s | 260 kB 00:00 2023-07-14T18:11:36.145 INFO:teuthology.orchestra.run.smithi144.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 472 kB/s | 165 kB 00:00 2023-07-14T18:11:36.149 INFO:teuthology.orchestra.run.smithi105.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T18:11:36.220 INFO:teuthology.orchestra.run.smithi144.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 239 kB/s | 18 kB 00:00 2023-07-14T18:11:36.562 INFO:teuthology.orchestra.run.smithi144.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 897 kB/s | 306 kB 00:00 2023-07-14T18:11:36.641 INFO:teuthology.orchestra.run.smithi144.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 746 kB/s | 59 kB 00:00 2023-07-14T18:11:36.696 INFO:teuthology.orchestra.run.smithi144.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-07-14T18:11:36.821 INFO:teuthology.orchestra.run.smithi144.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.7 MB/s | 476 kB 00:00 2023-07-14T18:11:36.872 INFO:teuthology.orchestra.run.smithi144.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 875 kB/s | 44 kB 00:00 2023-07-14T18:11:36.947 INFO:teuthology.orchestra.run.smithi144.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.4 MB/s | 185 kB 00:00 2023-07-14T18:11:36.997 INFO:teuthology.orchestra.run.smithi144.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 660 kB/s | 33 kB 00:00 2023-07-14T18:11:37.048 INFO:teuthology.orchestra.run.smithi144.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 327 kB/s | 16 kB 00:00 2023-07-14T18:11:37.161 INFO:teuthology.orchestra.run.smithi144.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-07-14T18:11:37.273 INFO:teuthology.orchestra.run.smithi144.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.4 MB/s | 384 kB 00:00 2023-07-14T18:11:37.324 INFO:teuthology.orchestra.run.smithi144.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-07-14T18:11:37.374 INFO:teuthology.orchestra.run.smithi144.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-07-14T18:11:37.425 INFO:teuthology.orchestra.run.smithi144.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-07-14T18:11:37.499 INFO:teuthology.orchestra.run.smithi105.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 572 kB/s | 772 kB 00:01 2023-07-14T18:11:37.600 INFO:teuthology.orchestra.run.smithi144.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 6.7 MB/s | 1.2 MB 00:00 2023-07-14T18:11:37.733 INFO:teuthology.orchestra.run.smithi105.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 580 kB/s | 135 kB 00:00 2023-07-14T18:11:37.861 INFO:teuthology.orchestra.run.smithi144.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 187 kB/s | 49 kB 00:00 2023-07-14T18:11:37.926 INFO:teuthology.orchestra.run.smithi144.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 909 kB/s | 59 kB 00:00 2023-07-14T18:11:37.968 INFO:teuthology.orchestra.run.smithi144.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-07-14T18:11:38.025 INFO:teuthology.orchestra.run.smithi105.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 477 kB/s | 139 kB 00:00 2023-07-14T18:11:38.029 INFO:teuthology.orchestra.run.smithi064.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 51 MB/s | 47 MB 00:00 2023-07-14T18:11:38.060 INFO:teuthology.orchestra.run.smithi144.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.0 MB/s | 283 kB 00:00 2023-07-14T18:11:38.102 INFO:teuthology.orchestra.run.smithi144.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 374 kB/s | 16 kB 00:00 2023-07-14T18:11:38.152 INFO:teuthology.orchestra.run.smithi144.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 672 kB/s | 34 kB 00:00 2023-07-14T18:11:38.244 INFO:teuthology.orchestra.run.smithi144.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-07-14T18:11:38.286 INFO:teuthology.orchestra.run.smithi144.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2023-07-14T18:11:38.328 INFO:teuthology.orchestra.run.smithi144.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 451 kB/s | 19 kB 00:00 2023-07-14T18:11:38.378 INFO:teuthology.orchestra.run.smithi144.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 478 kB/s | 24 kB 00:00 2023-07-14T18:11:38.420 INFO:teuthology.orchestra.run.smithi144.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 701 kB/s | 29 kB 00:00 2023-07-14T18:11:38.462 INFO:teuthology.orchestra.run.smithi144.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 661 kB/s | 27 kB 00:00 2023-07-14T18:11:38.546 INFO:teuthology.orchestra.run.smithi144.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 3.0 MB/s | 255 kB 00:00 2023-07-14T18:11:38.638 INFO:teuthology.orchestra.run.smithi144.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.7 MB/s | 251 kB 00:00 2023-07-14T18:11:38.688 INFO:teuthology.orchestra.run.smithi144.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-07-14T18:11:38.755 INFO:teuthology.orchestra.run.smithi144.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 1.3 MB/s | 85 kB 00:00 2023-07-14T18:11:38.814 INFO:teuthology.orchestra.run.smithi144.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 394 kB/s | 23 kB 00:00 2023-07-14T18:11:38.964 INFO:teuthology.orchestra.run.smithi144.stdout:(68/131): ceph-mgr-diskprediction-local-18.0.0- 1.7 MB/s | 7.4 MB 00:04 2023-07-14T18:11:38.975 INFO:teuthology.orchestra.run.smithi105.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 498 kB/s | 473 kB 00:00 2023-07-14T18:11:38.990 INFO:teuthology.orchestra.run.smithi144.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.8 MB/s | 70 kB 00:00 2023-07-14T18:11:39.140 INFO:teuthology.orchestra.run.smithi144.stdout:(70/131): thrift-0.13.0-2.el8.x86_64.rpm 5.3 MB/s | 1.7 MB 00:00 2023-07-14T18:11:39.198 INFO:teuthology.orchestra.run.smithi144.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.9 MB/s | 110 kB 00:00 2023-07-14T18:11:39.266 INFO:teuthology.orchestra.run.smithi144.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 811 kB/s | 54 kB 00:00 2023-07-14T18:11:39.408 INFO:teuthology.orchestra.run.smithi144.stdout:(73/131): python3-scipy-1.0.0-21.module+el8.5.0 33 MB/s | 14 MB 00:00 2023-07-14T18:11:39.433 INFO:teuthology.orchestra.run.smithi144.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.5 MB/s | 259 kB 00:00 2023-07-14T18:11:39.458 INFO:teuthology.orchestra.run.smithi144.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-14T18:11:39.483 INFO:teuthology.orchestra.run.smithi144.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 772 kB/s | 39 kB 00:00 2023-07-14T18:11:39.517 INFO:teuthology.orchestra.run.smithi144.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-07-14T18:11:39.551 INFO:teuthology.orchestra.run.smithi144.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.8 MB/s | 538 kB 00:00 2023-07-14T18:11:39.576 INFO:teuthology.orchestra.run.smithi144.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-14T18:11:39.601 INFO:teuthology.orchestra.run.smithi144.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-14T18:11:39.635 INFO:teuthology.orchestra.run.smithi144.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-14T18:11:39.668 INFO:teuthology.orchestra.run.smithi144.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 801 kB/s | 27 kB 00:00 2023-07-14T18:11:39.710 INFO:teuthology.orchestra.run.smithi144.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-14T18:11:39.777 INFO:teuthology.orchestra.run.smithi144.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 661 kB/s | 44 kB 00:00 2023-07-14T18:11:39.836 INFO:teuthology.orchestra.run.smithi144.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-14T18:11:39.886 INFO:teuthology.orchestra.run.smithi144.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 8.9 MB/s | 457 kB 00:00 2023-07-14T18:11:39.928 INFO:teuthology.orchestra.run.smithi144.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-07-14T18:11:39.987 INFO:teuthology.orchestra.run.smithi144.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 18 MB/s | 3.7 MB 00:00 2023-07-14T18:11:40.012 INFO:teuthology.orchestra.run.smithi144.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 767 kB/s | 64 kB 00:00 2023-07-14T18:11:40.037 INFO:teuthology.orchestra.run.smithi144.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.4 MB/s | 126 kB 00:00 2023-07-14T18:11:40.054 INFO:teuthology.orchestra.run.smithi144.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 475 kB/s | 20 kB 00:00 2023-07-14T18:11:40.080 INFO:teuthology.orchestra.run.smithi144.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2023-07-14T18:11:40.122 INFO:teuthology.orchestra.run.smithi144.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-07-14T18:11:40.164 INFO:teuthology.orchestra.run.smithi144.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 4.6 MB/s | 198 kB 00:00 2023-07-14T18:11:40.275 INFO:teuthology.orchestra.run.smithi144.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 25 MB/s | 4.9 MB 00:00 2023-07-14T18:11:40.323 INFO:teuthology.orchestra.run.smithi144.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.2 MB/s | 202 kB 00:00 2023-07-14T18:11:40.398 INFO:teuthology.orchestra.run.smithi144.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-07-14T18:11:40.423 INFO:teuthology.orchestra.run.smithi144.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.5 MB/s | 354 kB 00:00 2023-07-14T18:11:40.473 INFO:teuthology.orchestra.run.smithi144.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 16 MB/s | 808 kB 00:00 2023-07-14T18:11:40.566 INFO:teuthology.orchestra.run.smithi144.stdout:(100/131): postgresql-server-10.21-2.module+el8 30 MB/s | 5.1 MB 00:00 2023-07-14T18:11:40.608 INFO:teuthology.orchestra.run.smithi144.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-14T18:11:40.641 INFO:teuthology.orchestra.run.smithi144.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 623 kB/s | 47 kB 00:00 2023-07-14T18:11:40.667 INFO:teuthology.orchestra.run.smithi144.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 3.3 MB/s | 200 kB 00:00 2023-07-14T18:11:40.692 INFO:teuthology.orchestra.run.smithi144.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 846 kB/s | 43 kB 00:00 2023-07-14T18:11:40.717 INFO:teuthology.orchestra.run.smithi144.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-07-14T18:11:40.742 INFO:teuthology.orchestra.run.smithi144.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 956 kB/s | 48 kB 00:00 2023-07-14T18:11:40.787 INFO:teuthology.orchestra.run.smithi144.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.2 MB/s | 87 kB 00:00 2023-07-14T18:11:40.818 INFO:teuthology.orchestra.run.smithi144.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 566 kB/s | 43 kB 00:00 2023-07-14T18:11:40.843 INFO:teuthology.orchestra.run.smithi144.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.9 MB/s | 109 kB 00:00 2023-07-14T18:11:40.869 INFO:teuthology.orchestra.run.smithi144.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-07-14T18:11:40.902 INFO:teuthology.orchestra.run.smithi144.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-07-14T18:11:40.928 INFO:teuthology.orchestra.run.smithi144.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 319 kB/s | 19 kB 00:00 2023-07-14T18:11:40.953 INFO:teuthology.orchestra.run.smithi144.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-07-14T18:11:40.987 INFO:teuthology.orchestra.run.smithi144.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-07-14T18:11:41.012 INFO:teuthology.orchestra.run.smithi144.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 305 kB/s | 18 kB 00:00 2023-07-14T18:11:41.054 INFO:teuthology.orchestra.run.smithi144.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 8.8 MB/s | 605 kB 00:00 2023-07-14T18:11:41.071 INFO:teuthology.orchestra.run.smithi144.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-14T18:11:41.113 INFO:teuthology.orchestra.run.smithi144.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.7 MB/s | 72 kB 00:00 2023-07-14T18:11:41.155 INFO:teuthology.orchestra.run.smithi144.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 23 MB/s | 2.3 MB 00:00 2023-07-14T18:11:41.180 INFO:teuthology.orchestra.run.smithi144.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-07-14T18:11:41.205 INFO:teuthology.orchestra.run.smithi144.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-07-14T18:11:41.230 INFO:teuthology.orchestra.run.smithi144.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-14T18:11:41.256 INFO:teuthology.orchestra.run.smithi144.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-07-14T18:11:41.301 INFO:teuthology.orchestra.run.smithi144.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 549 kB/s | 39 kB 00:00 2023-07-14T18:11:41.340 INFO:teuthology.orchestra.run.smithi144.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 7.5 MB/s | 644 kB 00:00 2023-07-14T18:11:41.365 INFO:teuthology.orchestra.run.smithi144.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.6 MB/s | 171 kB 00:00 2023-07-14T18:11:41.382 INFO:teuthology.orchestra.run.smithi144.stdout:(127/131): NetworkManager-initscripts-updown-1. 3.2 MB/s | 138 kB 00:00 2023-07-14T18:11:44.057 INFO:teuthology.orchestra.run.smithi144.stdout:(128/131): librados2-18.0.0-4940.g81c9387d.el8. 1.4 MB/s | 3.7 MB 00:02 2023-07-14T18:11:44.116 INFO:teuthology.orchestra.run.smithi144.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-14T18:11:45.110 INFO:teuthology.orchestra.run.smithi105.stdout:(17/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 993 kB/s | 14 MB 00:13 2023-07-14T18:11:45.183 INFO:teuthology.orchestra.run.smithi144.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 978 kB/s | 3.6 MB 00:03 2023-07-14T18:11:45.210 INFO:teuthology.orchestra.run.smithi105.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 449 kB/s | 45 kB 00:00 2023-07-14T18:11:45.402 INFO:teuthology.orchestra.run.smithi105.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 640 kB/s | 122 kB 00:00 2023-07-14T18:11:45.669 INFO:teuthology.orchestra.run.smithi105.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 738 kB/s | 197 kB 00:00 2023-07-14T18:11:46.119 INFO:teuthology.orchestra.run.smithi105.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 825 kB/s | 371 kB 00:00 2023-07-14T18:11:46.511 INFO:teuthology.orchestra.run.smithi105.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 930 kB/s | 364 kB 00:00 2023-07-14T18:11:46.611 INFO:teuthology.orchestra.run.smithi105.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 1.0 MB/s | 107 kB 00:00 2023-07-14T18:11:46.728 INFO:teuthology.orchestra.run.smithi105.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 802 kB/s | 94 kB 00:00 2023-07-14T18:11:47.645 INFO:teuthology.orchestra.run.smithi105.stdout:(25/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 801 kB/s | 6.8 MB 00:08 2023-07-14T18:11:47.837 INFO:teuthology.orchestra.run.smithi105.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 934 kB/s | 179 kB 00:00 2023-07-14T18:11:47.912 INFO:teuthology.orchestra.run.smithi105.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 308 kB/s | 23 kB 00:00 2023-07-14T18:11:48.046 INFO:teuthology.orchestra.run.smithi105.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 970 kB/s | 129 kB 00:00 2023-07-14T18:11:48.980 INFO:teuthology.orchestra.run.smithi105.stdout:(29/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.6 MB/s | 3.5 MB 00:02 2023-07-14T18:11:49.421 INFO:teuthology.orchestra.run.smithi105.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.2 MB/s | 1.7 MB 00:01 2023-07-14T18:11:49.645 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:49.646 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:11:49.662 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:49.663 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:11:49.663 INFO:teuthology.orchestra.run.smithi105.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 996 kB/s | 241 kB 00:00 2023-07-14T18:11:49.664 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:11:49.666 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:11:49.668 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:11:49.671 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:11:49.673 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:11:49.755 INFO:teuthology.orchestra.run.smithi105.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 519 kB/s | 47 kB 00:00 2023-07-14T18:11:49.819 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2023-07-14T18:11:49.831 INFO:teuthology.orchestra.run.smithi105.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 193 kB/s | 15 kB 00:00 2023-07-14T18:11:49.832 INFO:teuthology.orchestra.run.smithi064.stdout:=========================================================================================================================== 2023-07-14T18:11:49.832 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2023-07-14T18:11:49.833 INFO:teuthology.orchestra.run.smithi064.stdout:=========================================================================================================================== 2023-07-14T18:11:49.833 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2023-07-14T18:11:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:11:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:11:49.833 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:11:49.834 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:11:49.835 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:11:49.836 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:11:49.837 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:11:49.838 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:11:49.839 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:11:49.840 INFO:teuthology.orchestra.run.smithi064.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:11:49.841 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:11:49.842 INFO:teuthology.orchestra.run.smithi064.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:11:49.843 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:11:49.844 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:11:49.845 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:11:49.846 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:11:49.847 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:11:49.848 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:11:49.849 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:11:49.850 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling module streams: 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql 10 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout:=========================================================================================================================== 2023-07-14T18:11:49.851 INFO:teuthology.orchestra.run.smithi064.stdout:Install 128 Packages 2023-07-14T18:11:49.852 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 3 Packages 2023-07-14T18:11:49.852 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:11:49.852 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 206 M 2023-07-14T18:11:49.852 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2023-07-14T18:11:50.064 INFO:teuthology.orchestra.run.smithi105.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 1.1 MB/s | 258 kB 00:00 2023-07-14T18:11:50.221 INFO:teuthology.orchestra.run.smithi064.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 27 kB/s | 6.4 kB 00:00 2023-07-14T18:11:50.239 INFO:teuthology.orchestra.run.smithi105.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 945 kB/s | 165 kB 00:00 2023-07-14T18:11:50.323 INFO:teuthology.orchestra.run.smithi105.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 215 kB/s | 18 kB 00:00 2023-07-14T18:11:51.046 INFO:teuthology.orchestra.run.smithi064.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.0 MB/s | 850 kB 00:00 2023-07-14T18:11:51.188 INFO:teuthology.orchestra.run.smithi064.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.2 MB/s | 167 kB 00:00 2023-07-14T18:11:51.249 INFO:teuthology.orchestra.run.smithi105.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 331 kB/s | 306 kB 00:00 2023-07-14T18:11:51.374 INFO:teuthology.orchestra.run.smithi105.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 471 kB/s | 59 kB 00:00 2023-07-14T18:11:51.533 INFO:teuthology.orchestra.run.smithi105.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 474 kB/s | 75 kB 00:00 2023-07-14T18:11:52.091 INFO:teuthology.orchestra.run.smithi105.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 852 kB/s | 476 kB 00:00 2023-07-14T18:11:52.217 INFO:teuthology.orchestra.run.smithi105.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 349 kB/s | 44 kB 00:00 2023-07-14T18:11:52.496 INFO:teuthology.orchestra.run.smithi064.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 1.8 MB/s | 2.3 MB 00:01 2023-07-14T18:11:52.600 INFO:teuthology.orchestra.run.smithi105.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 483 kB/s | 185 kB 00:00 2023-07-14T18:11:52.750 INFO:teuthology.orchestra.run.smithi105.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 220 kB/s | 33 kB 00:00 2023-07-14T18:11:52.893 INFO:teuthology.orchestra.run.smithi105.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 115 kB/s | 16 kB 00:00 2023-07-14T18:11:53.160 INFO:teuthology.orchestra.run.smithi105.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 647 kB/s | 173 kB 00:00 2023-07-14T18:11:53.818 INFO:teuthology.orchestra.run.smithi105.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 584 kB/s | 384 kB 00:00 2023-07-14T18:11:54.060 INFO:teuthology.orchestra.run.smithi105.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 336 kB/s | 81 kB 00:00 2023-07-14T18:11:54.188 INFO:teuthology.orchestra.run.smithi064.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 939 kB/s | 1.6 MB 00:01 2023-07-14T18:11:54.652 INFO:teuthology.orchestra.run.smithi105.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 19 kB/s | 11 kB 00:00 2023-07-14T18:11:54.761 INFO:teuthology.orchestra.run.smithi105.stdout:(49/131): ceph-mgr-diskprediction-local-18.0.0- 1.3 MB/s | 7.4 MB 00:05 2023-07-14T18:11:54.811 INFO:teuthology.orchestra.run.smithi105.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 111 kB/s | 18 kB 00:00 2023-07-14T18:11:54.953 INFO:teuthology.orchestra.run.smithi105.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 344 kB/s | 49 kB 00:00 2023-07-14T18:11:55.328 INFO:teuthology.orchestra.run.smithi105.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 157 kB/s | 59 kB 00:00 2023-07-14T18:11:55.595 INFO:teuthology.orchestra.run.smithi105.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 226 kB/s | 60 kB 00:00 2023-07-14T18:11:56.362 INFO:teuthology.orchestra.run.smithi105.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 369 kB/s | 283 kB 00:00 2023-07-14T18:11:56.412 INFO:teuthology.orchestra.run.smithi105.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 723 kB/s | 1.2 MB 00:01 2023-07-14T18:11:56.563 INFO:teuthology.orchestra.run.smithi105.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 224 kB/s | 34 kB 00:00 2023-07-14T18:11:56.780 INFO:teuthology.orchestra.run.smithi105.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 907 kB/s | 196 kB 00:00 2023-07-14T18:11:56.897 INFO:teuthology.orchestra.run.smithi105.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 527 kB/s | 61 kB 00:00 2023-07-14T18:11:56.989 INFO:teuthology.orchestra.run.smithi105.stdout:(59/131): python3-portend-2.6-1.el8.noarch.rpm 25 kB/s | 16 kB 00:00 2023-07-14T18:11:57.014 INFO:teuthology.orchestra.run.smithi105.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 160 kB/s | 19 kB 00:00 2023-07-14T18:11:57.106 INFO:teuthology.orchestra.run.smithi105.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 204 kB/s | 24 kB 00:00 2023-07-14T18:11:57.131 INFO:teuthology.orchestra.run.smithi105.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 249 kB/s | 29 kB 00:00 2023-07-14T18:11:57.240 INFO:teuthology.orchestra.run.smithi105.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 206 kB/s | 27 kB 00:00 2023-07-14T18:11:57.440 INFO:teuthology.orchestra.run.smithi105.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 828 kB/s | 255 kB 00:00 2023-07-14T18:11:57.557 INFO:teuthology.orchestra.run.smithi105.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 525 kB/s | 61 kB 00:00 2023-07-14T18:11:57.666 INFO:teuthology.orchestra.run.smithi105.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 789 kB/s | 85 kB 00:00 2023-07-14T18:11:57.775 INFO:teuthology.orchestra.run.smithi105.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 212 kB/s | 23 kB 00:00 2023-07-14T18:11:57.992 INFO:teuthology.orchestra.run.smithi105.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 334 kB/s | 251 kB 00:00 2023-07-14T18:11:58.017 INFO:teuthology.orchestra.run.smithi105.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T18:11:58.030 INFO:teuthology.orchestra.run.smithi064.stdout:(6/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 684 kB/s | 5.4 MB 00:08 2023-07-14T18:11:58.075 INFO:teuthology.orchestra.run.smithi105.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 803 kB/s | 47 kB 00:00 2023-07-14T18:11:58.117 INFO:teuthology.orchestra.run.smithi105.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-14T18:11:58.160 INFO:teuthology.orchestra.run.smithi105.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-14T18:11:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-14T18:11:58.235 INFO:teuthology.orchestra.run.smithi105.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-07-14T18:11:58.277 INFO:teuthology.orchestra.run.smithi105.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.1 MB/s | 87 kB 00:00 2023-07-14T18:11:58.302 INFO:teuthology.orchestra.run.smithi105.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2023-07-14T18:11:58.344 INFO:teuthology.orchestra.run.smithi105.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 2.6 MB/s | 109 kB 00:00 2023-07-14T18:11:58.387 INFO:teuthology.orchestra.run.smithi105.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.4 MB/s | 229 kB 00:00 2023-07-14T18:11:58.428 INFO:teuthology.orchestra.run.smithi105.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 13 MB/s | 560 kB 00:00 2023-07-14T18:11:58.456 INFO:teuthology.orchestra.run.smithi105.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 747 kB/s | 19 kB 00:00 2023-07-14T18:11:58.495 INFO:teuthology.orchestra.run.smithi105.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-14T18:11:58.537 INFO:teuthology.orchestra.run.smithi105.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 13 MB/s | 559 kB 00:00 2023-07-14T18:11:58.571 INFO:teuthology.orchestra.run.smithi105.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 540 kB/s | 18 kB 00:00 2023-07-14T18:11:58.613 INFO:teuthology.orchestra.run.smithi105.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 14 MB/s | 605 kB 00:00 2023-07-14T18:11:58.655 INFO:teuthology.orchestra.run.smithi105.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-14T18:11:58.738 INFO:teuthology.orchestra.run.smithi105.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-14T18:11:58.780 INFO:teuthology.orchestra.run.smithi105.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-07-14T18:11:58.814 INFO:teuthology.orchestra.run.smithi105.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 2.4 MB/s | 83 kB 00:00 2023-07-14T18:11:58.848 INFO:teuthology.orchestra.run.smithi105.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.7 MB/s | 60 kB 00:00 2023-07-14T18:11:58.890 INFO:teuthology.orchestra.run.smithi105.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-07-14T18:11:58.924 INFO:teuthology.orchestra.run.smithi105.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.7 MB/s | 58 kB 00:00 2023-07-14T18:11:58.957 INFO:teuthology.orchestra.run.smithi105.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-07-14T18:11:58.999 INFO:teuthology.orchestra.run.smithi105.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 15 MB/s | 644 kB 00:00 2023-07-14T18:11:59.041 INFO:teuthology.orchestra.run.smithi105.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-14T18:11:59.066 INFO:teuthology.orchestra.run.smithi105.stdout:(95/131): NetworkManager-initscripts-updown-1.3 5.4 MB/s | 138 kB 00:00 2023-07-14T18:11:59.225 INFO:teuthology.orchestra.run.smithi105.stdout:(96/131): thrift-0.13.0-2.el8.x86_64.rpm 1.2 MB/s | 1.7 MB 00:01 2023-07-14T18:11:59.392 INFO:teuthology.orchestra.run.smithi105.stdout:(97/131): python3-scipy-1.0.0-21.module+el8.5.0 42 MB/s | 14 MB 00:00 2023-07-14T18:11:59.417 INFO:teuthology.orchestra.run.smithi105.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 574 kB/s | 110 kB 00:00 2023-07-14T18:11:59.442 INFO:teuthology.orchestra.run.smithi105.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-07-14T18:11:59.468 INFO:teuthology.orchestra.run.smithi105.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-07-14T18:11:59.493 INFO:teuthology.orchestra.run.smithi105.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-14T18:11:59.518 INFO:teuthology.orchestra.run.smithi105.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 771 kB/s | 39 kB 00:00 2023-07-14T18:11:59.560 INFO:teuthology.orchestra.run.smithi105.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 10 MB/s | 688 kB 00:00 2023-07-14T18:11:59.594 INFO:teuthology.orchestra.run.smithi105.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.0 MB/s | 538 kB 00:00 2023-07-14T18:11:59.619 INFO:teuthology.orchestra.run.smithi105.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-14T18:11:59.644 INFO:teuthology.orchestra.run.smithi105.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-14T18:11:59.678 INFO:teuthology.orchestra.run.smithi105.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-07-14T18:11:59.703 INFO:teuthology.orchestra.run.smithi105.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 1.0 MB/s | 27 kB 00:00 2023-07-14T18:11:59.745 INFO:teuthology.orchestra.run.smithi105.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-14T18:11:59.770 INFO:teuthology.orchestra.run.smithi105.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.7 MB/s | 44 kB 00:00 2023-07-14T18:11:59.837 INFO:teuthology.orchestra.run.smithi105.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-07-14T18:11:59.879 INFO:teuthology.orchestra.run.smithi105.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2023-07-14T18:11:59.921 INFO:teuthology.orchestra.run.smithi105.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-14T18:11:59.988 INFO:teuthology.orchestra.run.smithi105.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2023-07-14T18:12:00.013 INFO:teuthology.orchestra.run.smithi105.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 696 kB/s | 64 kB 00:00 2023-07-14T18:12:00.039 INFO:teuthology.orchestra.run.smithi105.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-14T18:12:00.064 INFO:teuthology.orchestra.run.smithi105.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 397 kB/s | 20 kB 00:00 2023-07-14T18:12:00.106 INFO:teuthology.orchestra.run.smithi105.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.5 MB/s | 103 kB 00:00 2023-07-14T18:12:00.140 INFO:teuthology.orchestra.run.smithi105.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 9.2 MB/s | 709 kB 00:00 2023-07-14T18:12:00.174 INFO:teuthology.orchestra.run.smithi105.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 5.8 MB/s | 198 kB 00:00 2023-07-14T18:12:00.266 INFO:teuthology.orchestra.run.smithi105.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 31 MB/s | 4.9 MB 00:00 2023-07-14T18:12:00.299 INFO:teuthology.orchestra.run.smithi105.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 6.0 MB/s | 202 kB 00:00 2023-07-14T18:12:00.366 INFO:teuthology.orchestra.run.smithi105.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2023-07-14T18:12:00.391 INFO:teuthology.orchestra.run.smithi105.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2023-07-14T18:12:00.450 INFO:teuthology.orchestra.run.smithi105.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-14T18:12:00.542 INFO:teuthology.orchestra.run.smithi105.stdout:(126/131): postgresql-server-10.21-2.module+el8 29 MB/s | 5.1 MB 00:00 2023-07-14T18:12:00.584 INFO:teuthology.orchestra.run.smithi105.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-14T18:12:01.821 INFO:teuthology.orchestra.run.smithi064.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 620 kB/s | 4.6 MB 00:07 2023-07-14T18:12:02.667 INFO:teuthology.orchestra.run.smithi144.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 1.5 MB/s | 48 MB 00:31 2023-07-14T18:12:02.670 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:12:02.670 INFO:teuthology.orchestra.run.smithi144.stdout:Total 5.6 MB/s | 206 MB 00:36 2023-07-14T18:12:03.401 INFO:teuthology.orchestra.run.smithi105.stdout:(128/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 3.6 MB 00:02 2023-07-14T18:12:03.451 INFO:teuthology.orchestra.run.smithi105.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-14T18:12:03.471 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2023-07-14T18:12:03.591 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2023-07-14T18:12:03.591 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2023-07-14T18:12:04.718 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2023-07-14T18:12:04.723 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2023-07-14T18:12:06.093 INFO:teuthology.orchestra.run.smithi105.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 679 kB/s | 3.7 MB 00:05 2023-07-14T18:12:09.575 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2023-07-14T18:12:09.780 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:12:09.950 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:12:10.154 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T18:12:10.348 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:12:10.472 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:12:10.764 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:12:11.163 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:12:11.397 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:12:11.734 INFO:teuthology.orchestra.run.smithi105.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 1.3 MB/s | 48 MB 00:38 2023-07-14T18:12:11.737 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:12:11.737 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.6 MB/s | 206 MB 00:44 2023-07-14T18:12:11.796 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:12:11.989 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:12:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:12:12.322 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:12:12.423 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:12:12.552 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2023-07-14T18:12:12.581 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:12:12.673 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2023-07-14T18:12:12.674 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2023-07-14T18:12:12.851 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:12:13.050 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:12:13.424 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:12:13.670 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:12:13.788 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2023-07-14T18:12:13.792 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2023-07-14T18:12:13.830 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:12:13.944 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:12:14.097 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:12:14.165 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:12:14.321 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:12:14.472 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:12:14.638 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:12:15.249 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:12:15.455 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:12:16.049 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:12:16.238 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:12:16.965 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:12:17.203 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T18:12:17.446 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T18:12:17.658 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T18:12:17.835 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T18:12:18.068 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T18:12:18.289 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:12:18.494 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2023-07-14T18:12:18.494 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:12:18.640 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:12:18.661 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:12:18.853 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:12:19.030 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:12:19.144 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:12:19.355 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:12:19.565 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:12:19.756 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:12:19.868 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T18:12:19.971 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:12:20.206 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T18:12:20.371 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:12:20.445 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T18:12:20.555 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:12:20.662 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:12:20.667 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:12:20.872 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:12:20.956 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:12:21.110 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:12:21.280 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T18:12:21.484 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:12:21.691 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:12:21.887 INFO:teuthology.orchestra.run.smithi064.stdout:(8/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 762 kB/s | 18 MB 00:23 2023-07-14T18:12:22.034 INFO:teuthology.orchestra.run.smithi064.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 165 kB/s | 24 kB 00:00 2023-07-14T18:12:22.062 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:12:22.283 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:12:22.416 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:12:22.429 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:12:22.542 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:12:22.696 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:12:22.772 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T18:12:22.845 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:12:22.981 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:12:23.050 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T18:12:23.197 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:12:23.216 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T18:12:23.367 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:12:23.370 INFO:teuthology.orchestra.run.smithi064.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 643 kB/s | 14 MB 00:21 2023-07-14T18:12:23.375 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T18:12:23.512 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:12:23.582 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:12:23.671 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:12:23.719 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T18:12:23.821 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:12:23.871 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T18:12:23.921 INFO:teuthology.orchestra.run.smithi064.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 473 kB/s | 260 kB 00:00 2023-07-14T18:12:24.038 INFO:teuthology.orchestra.run.smithi064.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T18:12:24.039 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T18:12:24.207 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T18:12:24.419 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T18:12:24.428 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:12:24.588 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:12:24.651 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T18:12:25.021 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T18:12:25.168 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:12:25.315 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T18:12:25.329 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:12:25.422 INFO:teuthology.orchestra.run.smithi064.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 558 kB/s | 772 kB 00:01 2023-07-14T18:12:25.488 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T18:12:25.714 INFO:teuthology.orchestra.run.smithi064.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 465 kB/s | 135 kB 00:00 2023-07-14T18:12:25.723 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T18:12:25.958 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T18:12:25.980 INFO:teuthology.orchestra.run.smithi064.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 522 kB/s | 139 kB 00:00 2023-07-14T18:12:26.007 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:12:26.278 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T18:12:26.311 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T18:12:26.469 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T18:12:26.545 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T18:12:26.697 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:12:26.705 INFO:teuthology.orchestra.run.smithi064.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 652 kB/s | 473 kB 00:00 2023-07-14T18:12:26.754 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T18:12:26.877 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:12:26.966 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T18:12:27.007 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:12:27.132 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T18:12:27.224 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:12:27.323 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T18:12:27.429 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:12:27.605 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T18:12:27.748 INFO:teuthology.orchestra.run.smithi064.stdout:(17/131): ceph-common-18.0.0-4940.g81c9387d.el8 601 kB/s | 22 MB 00:37 2023-07-14T18:12:27.865 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T18:12:27.973 INFO:teuthology.orchestra.run.smithi064.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 199 kB/s | 45 kB 00:00 2023-07-14T18:12:28.098 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T18:12:28.343 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T18:12:28.365 INFO:teuthology.orchestra.run.smithi064.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 313 kB/s | 122 kB 00:00 2023-07-14T18:12:28.502 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T18:12:28.646 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T18:12:28.831 INFO:teuthology.orchestra.run.smithi064.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 422 kB/s | 197 kB 00:00 2023-07-14T18:12:28.844 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:12:28.913 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:12:29.155 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T18:12:29.189 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:12:29.390 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T18:12:29.553 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:12:29.641 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:12:29.873 INFO:teuthology.orchestra.run.smithi064.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 356 kB/s | 371 kB 00:01 2023-07-14T18:12:29.900 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T18:12:30.072 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T18:12:30.197 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T18:12:30.220 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:12:30.442 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:12:30.527 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T18:12:30.702 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:12:30.789 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T18:12:30.832 INFO:teuthology.orchestra.run.smithi064.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 380 kB/s | 364 kB 00:00 2023-07-14T18:12:30.935 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:12:30.996 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T18:12:31.099 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:12:31.149 INFO:teuthology.orchestra.run.smithi064.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 337 kB/s | 107 kB 00:00 2023-07-14T18:12:31.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:12:31.247 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:12:31.389 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:12:31.441 INFO:teuthology.orchestra.run.smithi064.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 321 kB/s | 94 kB 00:00 2023-07-14T18:12:31.480 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:12:31.559 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:12:31.682 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:12:31.752 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:12:31.856 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:12:31.911 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T18:12:32.072 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:12:32.081 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T18:12:32.225 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:12:32.262 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:12:32.337 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:12:32.489 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:12:32.554 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T18:12:32.723 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:12:32.782 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T18:12:32.925 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:12:32.928 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:12:33.007 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:12:33.165 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:12:33.200 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T18:12:33.349 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T18:12:33.427 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:12:33.527 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T18:12:33.610 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:12:33.706 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T18:12:33.836 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:12:33.856 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T18:12:34.007 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T18:12:34.212 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T18:12:34.426 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T18:12:34.799 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T18:12:34.960 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:12:35.072 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T18:12:35.289 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T18:12:35.497 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T18:12:35.684 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:12:35.707 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T18:12:35.954 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:12:36.001 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T18:12:36.219 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T18:12:36.321 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:12:36.412 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T18:12:36.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T18:12:36.652 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:12:36.807 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T18:12:36.868 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:12:36.965 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T18:12:37.273 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T18:12:37.549 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T18:12:37.682 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:12:37.791 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T18:12:37.843 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:12:37.918 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:12:38.029 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T18:12:38.102 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:12:38.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:12:38.340 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:12:38.586 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:12:38.801 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:12:38.957 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:12:39.104 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:12:39.294 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:12:39.514 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:12:39.721 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:12:39.889 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:12:40.026 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:12:40.245 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:12:40.465 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:12:40.634 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:12:40.883 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:12:40.991 INFO:teuthology.orchestra.run.smithi064.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 379 kB/s | 3.5 MB 00:09 2023-07-14T18:12:41.085 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:12:41.409 INFO:teuthology.orchestra.run.smithi064.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 428 kB/s | 179 kB 00:00 2023-07-14T18:12:41.509 INFO:teuthology.orchestra.run.smithi064.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 232 kB/s | 23 kB 00:00 2023-07-14T18:12:41.588 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:12:41.776 INFO:teuthology.orchestra.run.smithi064.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 485 kB/s | 129 kB 00:00 2023-07-14T18:12:41.789 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:41.803 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:12:41.993 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:42.572 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:42.896 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:12:42.910 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:12:42.981 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:12:43.126 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:12:43.198 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:12:43.452 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:12:43.743 INFO:teuthology.orchestra.run.smithi064.stdout:(29/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 408 kB/s | 6.8 MB 00:17 2023-07-14T18:12:43.759 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:12:44.028 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:12:44.219 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:12:44.726 INFO:teuthology.orchestra.run.smithi064.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 595 kB/s | 1.7 MB 00:02 2023-07-14T18:12:44.988 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:12:45.185 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:12:45.247 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:12:45.343 INFO:teuthology.orchestra.run.smithi064.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 390 kB/s | 241 kB 00:00 2023-07-14T18:12:45.394 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:12:45.543 INFO:teuthology.orchestra.run.smithi064.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 237 kB/s | 47 kB 00:00 2023-07-14T18:12:45.644 INFO:teuthology.orchestra.run.smithi064.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 146 kB/s | 15 kB 00:00 2023-07-14T18:12:46.369 INFO:teuthology.orchestra.run.smithi064.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 356 kB/s | 258 kB 00:00 2023-07-14T18:12:46.811 INFO:teuthology.orchestra.run.smithi064.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 374 kB/s | 165 kB 00:00 2023-07-14T18:12:46.919 INFO:teuthology.orchestra.run.smithi064.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 166 kB/s | 18 kB 00:00 2023-07-14T18:12:47.220 INFO:teuthology.orchestra.run.smithi064.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.0 MB/s | 306 kB 00:00 2023-07-14T18:12:47.278 INFO:teuthology.orchestra.run.smithi064.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-07-14T18:12:47.354 INFO:teuthology.orchestra.run.smithi064.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-07-14T18:12:47.512 INFO:teuthology.orchestra.run.smithi064.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.9 MB/s | 476 kB 00:00 2023-07-14T18:12:47.571 INFO:teuthology.orchestra.run.smithi064.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 748 kB/s | 44 kB 00:00 2023-07-14T18:12:47.638 INFO:teuthology.orchestra.run.smithi064.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-07-14T18:12:47.697 INFO:teuthology.orchestra.run.smithi064.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 568 kB/s | 33 kB 00:00 2023-07-14T18:12:47.755 INFO:teuthology.orchestra.run.smithi064.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 279 kB/s | 16 kB 00:00 2023-07-14T18:12:47.822 INFO:teuthology.orchestra.run.smithi064.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.5 MB/s | 173 kB 00:00 2023-07-14T18:12:47.948 INFO:teuthology.orchestra.run.smithi064.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.0 MB/s | 384 kB 00:00 2023-07-14T18:12:48.040 INFO:teuthology.orchestra.run.smithi064.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 887 kB/s | 81 kB 00:00 2023-07-14T18:12:48.107 INFO:teuthology.orchestra.run.smithi064.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 167 kB/s | 11 kB 00:00 2023-07-14T18:12:48.173 INFO:teuthology.orchestra.run.smithi064.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 264 kB/s | 18 kB 00:00 2023-07-14T18:12:48.516 INFO:teuthology.orchestra.run.smithi064.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2023-07-14T18:12:48.582 INFO:teuthology.orchestra.run.smithi064.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 732 kB/s | 49 kB 00:00 2023-07-14T18:12:48.649 INFO:teuthology.orchestra.run.smithi064.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 883 kB/s | 59 kB 00:00 2023-07-14T18:12:48.716 INFO:teuthology.orchestra.run.smithi064.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 904 kB/s | 60 kB 00:00 2023-07-14T18:12:48.841 INFO:teuthology.orchestra.run.smithi064.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2023-07-14T18:12:48.925 INFO:teuthology.orchestra.run.smithi064.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 187 kB/s | 16 kB 00:00 2023-07-14T18:12:49.009 INFO:teuthology.orchestra.run.smithi064.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 404 kB/s | 34 kB 00:00 2023-07-14T18:12:49.055 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:49.117 INFO:teuthology.orchestra.run.smithi064.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 1.8 MB/s | 196 kB 00:00 2023-07-14T18:12:49.184 INFO:teuthology.orchestra.run.smithi064.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 925 kB/s | 61 kB 00:00 2023-07-14T18:12:49.251 INFO:teuthology.orchestra.run.smithi064.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 281 kB/s | 19 kB 00:00 2023-07-14T18:12:49.269 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:49.318 INFO:teuthology.orchestra.run.smithi064.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 359 kB/s | 24 kB 00:00 2023-07-14T18:12:49.385 INFO:teuthology.orchestra.run.smithi064.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 440 kB/s | 29 kB 00:00 2023-07-14T18:12:49.644 INFO:teuthology.orchestra.run.smithi064.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 106 kB/s | 27 kB 00:00 2023-07-14T18:12:49.828 INFO:teuthology.orchestra.run.smithi064.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 1.4 MB/s | 255 kB 00:00 2023-07-14T18:12:49.849 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:12:49.987 INFO:teuthology.orchestra.run.smithi064.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.6 MB/s | 251 kB 00:00 2023-07-14T18:12:50.053 INFO:teuthology.orchestra.run.smithi064.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 917 kB/s | 61 kB 00:00 2023-07-14T18:12:50.137 INFO:teuthology.orchestra.run.smithi064.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-07-14T18:12:50.152 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:12:50.213 INFO:teuthology.orchestra.run.smithi064.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 306 kB/s | 23 kB 00:00 2023-07-14T18:12:50.232 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:12:50.376 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:12:51.204 INFO:teuthology.orchestra.run.smithi064.stdout:(68/131): thrift-0.13.0-2.el8.x86_64.rpm 1.7 MB/s | 1.7 MB 00:00 2023-07-14T18:12:51.230 INFO:teuthology.orchestra.run.smithi064.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T18:12:51.288 INFO:teuthology.orchestra.run.smithi064.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 804 kB/s | 47 kB 00:00 2023-07-14T18:12:51.330 INFO:teuthology.orchestra.run.smithi064.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 4.7 MB/s | 200 kB 00:00 2023-07-14T18:12:51.364 INFO:teuthology.orchestra.run.smithi064.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-07-14T18:12:51.406 INFO:teuthology.orchestra.run.smithi064.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-07-14T18:12:51.439 INFO:teuthology.orchestra.run.smithi064.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 1.4 MB/s | 48 kB 00:00 2023-07-14T18:12:51.481 INFO:teuthology.orchestra.run.smithi064.stdout:(75/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-14T18:12:51.515 INFO:teuthology.orchestra.run.smithi064.stdout:(76/131): python3-requests-oauthlib-1.0.0-1.el8 1.3 MB/s | 43 kB 00:00 2023-07-14T18:12:51.549 INFO:teuthology.orchestra.run.smithi064.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 3.2 MB/s | 109 kB 00:00 2023-07-14T18:12:51.640 INFO:teuthology.orchestra.run.smithi064.stdout:(78/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 2.4 MB/s | 229 kB 00:00 2023-07-14T18:12:51.691 INFO:teuthology.orchestra.run.smithi064.stdout:(79/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-07-14T18:12:51.716 INFO:teuthology.orchestra.run.smithi064.stdout:(80/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 752 kB/s | 19 kB 00:00 2023-07-14T18:12:51.758 INFO:teuthology.orchestra.run.smithi064.stdout:(81/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.6 MB/s | 155 kB 00:00 2023-07-14T18:12:51.808 INFO:teuthology.orchestra.run.smithi064.stdout:(82/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-07-14T18:12:51.850 INFO:teuthology.orchestra.run.smithi064.stdout:(83/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 431 kB/s | 18 kB 00:00 2023-07-14T18:12:51.901 INFO:teuthology.orchestra.run.smithi064.stdout:(84/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-14T18:12:51.943 INFO:teuthology.orchestra.run.smithi064.stdout:(85/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 5.6 MB/s | 238 kB 00:00 2023-07-14T18:12:52.026 INFO:teuthology.orchestra.run.smithi064.stdout:(86/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 28 MB/s | 2.3 MB 00:00 2023-07-14T18:12:52.077 INFO:teuthology.orchestra.run.smithi064.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2023-07-14T18:12:52.110 INFO:teuthology.orchestra.run.smithi064.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 2.5 MB/s | 83 kB 00:00 2023-07-14T18:12:52.152 INFO:teuthology.orchestra.run.smithi064.stdout:(89/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-07-14T18:12:52.194 INFO:teuthology.orchestra.run.smithi064.stdout:(90/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-07-14T18:12:52.236 INFO:teuthology.orchestra.run.smithi064.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2023-07-14T18:12:52.278 INFO:teuthology.orchestra.run.smithi064.stdout:(92/131): libcom_err-devel-1.45.6-4.el8.x86_64. 929 kB/s | 39 kB 00:00 2023-07-14T18:12:52.328 INFO:teuthology.orchestra.run.smithi064.stdout:(93/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 13 MB/s | 644 kB 00:00 2023-07-14T18:12:52.370 INFO:teuthology.orchestra.run.smithi064.stdout:(94/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 4.0 MB/s | 171 kB 00:00 2023-07-14T18:12:52.412 INFO:teuthology.orchestra.run.smithi064.stdout:(95/131): NetworkManager-initscripts-updown-1.3 3.2 MB/s | 138 kB 00:00 2023-07-14T18:12:52.713 INFO:teuthology.orchestra.run.smithi064.stdout:(96/131): python3-scipy-1.0.0-21.module+el8.5.0 46 MB/s | 14 MB 00:00 2023-07-14T18:12:52.805 INFO:teuthology.orchestra.run.smithi064.stdout:(97/131): ceph-mgr-diskprediction-local-18.0.0- 837 kB/s | 7.4 MB 00:09 2023-07-14T18:12:52.830 INFO:teuthology.orchestra.run.smithi064.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 943 kB/s | 110 kB 00:00 2023-07-14T18:12:52.880 INFO:teuthology.orchestra.run.smithi064.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 723 kB/s | 54 kB 00:00 2023-07-14T18:12:52.905 INFO:teuthology.orchestra.run.smithi064.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.8 MB/s | 259 kB 00:00 2023-07-14T18:12:52.931 INFO:teuthology.orchestra.run.smithi064.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-07-14T18:12:52.956 INFO:teuthology.orchestra.run.smithi064.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 769 kB/s | 39 kB 00:00 2023-07-14T18:12:52.990 INFO:teuthology.orchestra.run.smithi064.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-07-14T18:12:53.023 INFO:teuthology.orchestra.run.smithi064.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2023-07-14T18:12:53.048 INFO:teuthology.orchestra.run.smithi064.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-07-14T18:12:53.074 INFO:teuthology.orchestra.run.smithi064.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-07-14T18:12:53.116 INFO:teuthology.orchestra.run.smithi064.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-07-14T18:12:53.157 INFO:teuthology.orchestra.run.smithi064.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 637 kB/s | 27 kB 00:00 2023-07-14T18:12:53.199 INFO:teuthology.orchestra.run.smithi064.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-07-14T18:12:53.225 INFO:teuthology.orchestra.run.smithi064.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.7 MB/s | 44 kB 00:00 2023-07-14T18:12:53.300 INFO:teuthology.orchestra.run.smithi064.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-07-14T18:12:53.350 INFO:teuthology.orchestra.run.smithi064.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-14T18:12:53.409 INFO:teuthology.orchestra.run.smithi064.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 3.8 MB/s | 225 kB 00:00 2023-07-14T18:12:53.476 INFO:teuthology.orchestra.run.smithi064.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-07-14T18:12:53.501 INFO:teuthology.orchestra.run.smithi064.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 695 kB/s | 64 kB 00:00 2023-07-14T18:12:53.535 INFO:teuthology.orchestra.run.smithi064.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.1 MB/s | 126 kB 00:00 2023-07-14T18:12:53.560 INFO:teuthology.orchestra.run.smithi064.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 342 kB/s | 20 kB 00:00 2023-07-14T18:12:53.585 INFO:teuthology.orchestra.run.smithi064.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-07-14T18:12:53.619 INFO:teuthology.orchestra.run.smithi064.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-14T18:12:53.670 INFO:teuthology.orchestra.run.smithi064.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.9 MB/s | 198 kB 00:00 2023-07-14T18:12:53.778 INFO:teuthology.orchestra.run.smithi064.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 25 MB/s | 4.9 MB 00:00 2023-07-14T18:12:53.812 INFO:teuthology.orchestra.run.smithi064.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 6.0 MB/s | 202 kB 00:00 2023-07-14T18:12:53.904 INFO:teuthology.orchestra.run.smithi064.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2023-07-14T18:12:53.929 INFO:teuthology.orchestra.run.smithi064.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.0 MB/s | 354 kB 00:00 2023-07-14T18:12:53.996 INFO:teuthology.orchestra.run.smithi064.stdout:(125/131): postgresql-contrib-10.21-2.module+el 12 MB/s | 808 kB 00:00 2023-07-14T18:12:54.088 INFO:teuthology.orchestra.run.smithi064.stdout:(126/131): postgresql-server-10.21-2.module+el8 28 MB/s | 5.1 MB 00:00 2023-07-14T18:12:54.130 INFO:teuthology.orchestra.run.smithi064.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 11 MB/s | 1.5 MB 00:00 2023-07-14T18:12:56.880 INFO:teuthology.orchestra.run.smithi064.stdout:(128/131): librados2-18.0.0-4940.g81c9387d.el8. 1.3 MB/s | 3.7 MB 00:02 2023-07-14T18:12:56.930 INFO:teuthology.orchestra.run.smithi064.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-07-14T18:13:00.313 INFO:teuthology.orchestra.run.smithi064.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 601 kB/s | 3.6 MB 00:06 2023-07-14T18:13:02.755 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /sys 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /proc 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /dev 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /run 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /mnt 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /var/tmp 2023-07-14T18:13:02.756 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /home 2023-07-14T18:13:02.757 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /tmp 2023-07-14T18:13:02.757 INFO:teuthology.orchestra.run.smithi144.stdout:skipping the directory /dev 2023-07-14T18:13:02.757 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:13:03.073 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:13:04.674 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:13:04.851 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:13:05.073 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:13:05.273 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:13:05.730 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:13:05.914 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:13:05.931 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:13:07.728 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:13:07.887 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:13:08.011 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:13:08.211 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:13:08.421 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:13:08.639 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:13:08.950 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:13:09.086 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /run 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2023-07-14T18:13:09.087 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2023-07-14T18:13:09.088 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2023-07-14T18:13:09.088 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /dev 2023-07-14T18:13:09.088 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:13:09.433 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:13:09.910 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:13:10.063 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:13:10.154 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:13:10.545 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:13:10.758 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:13:10.793 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:13:10.971 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:13:10.982 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:13:11.011 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:13:11.199 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:13:11.418 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:13:11.574 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:13:12.029 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:13:12.191 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:13:12.208 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:13:12.284 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:13:12.489 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:13:12.525 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:13:12.663 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:13:12.705 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:13:12.855 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:13:13.207 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:13:13.363 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:13:13.822 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:13:13.993 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:13:14.114 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:13:14.305 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:13:14.499 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:13:14.715 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:13:14.999 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:13:15.959 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:13:16.132 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:13:16.221 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:13:16.593 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:13:16.818 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:13:16.853 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:13:17.007 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:13:17.018 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:13:17.170 INFO:teuthology.orchestra.run.smithi064.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 891 kB/s | 48 MB 00:55 2023-07-14T18:13:17.173 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:13:17.174 INFO:teuthology.orchestra.run.smithi064.stdout:Total 2.4 MB/s | 206 MB 01:27 2023-07-14T18:13:17.935 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:13:17.985 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2023-07-14T18:13:18.116 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2023-07-14T18:13:18.117 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2023-07-14T18:13:18.227 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:13:18.326 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:13:18.378 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:13:18.519 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:13:18.554 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:13:18.571 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:13:18.699 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:13:18.741 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:13:18.899 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:13:19.250 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:13:19.289 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2023-07-14T18:13:19.293 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2023-07-14T18:13:19.424 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:13:19.454 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T18:13:19.811 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T18:13:19.982 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T18:13:20.199 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:13:20.349 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:13:20.488 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:13:22.619 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:22.911 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:22.929 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:23.162 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:13:23.394 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:13:23.705 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:13:23.961 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:13:24.013 INFO:teuthology.orchestra.run.smithi144.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:13:24.144 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:13:24.152 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:13:24.156 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2023-07-14T18:13:24.315 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:13:24.359 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T18:13:24.492 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:13:24.519 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:13:24.604 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:13:24.668 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:13:24.812 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:13:24.822 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:13:24.827 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:13:24.840 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:13:24.840 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:13:25.030 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:25.030 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:13:25.129 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T18:13:25.300 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:13:25.317 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T18:13:25.445 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:13:25.501 INFO:teuthology.orchestra.run.smithi144.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:25.591 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T18:13:25.667 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:13:25.748 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:13:25.943 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:13:26.001 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:26.001 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:13:26.001 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:13:26.002 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:13:26.003 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:13:26.004 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:13:26.004 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:13:26.004 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:13:26.004 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:13:26.004 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:13:26.005 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:13:26.005 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:13:26.005 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:13:26.006 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:13:26.013 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:13:26.014 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:13:26.015 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:13:26.016 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:13:26.017 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T18:13:26.018 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T18:13:26.019 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T18:13:26.020 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T18:13:26.021 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T18:13:26.022 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T18:13:26.023 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T18:13:26.024 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:13:26.025 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:26.026 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:13:26.087 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:13:26.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T18:13:26.251 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:13:26.335 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:13:26.382 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:13:26.490 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:13:26.684 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:13:26.786 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:13:26.947 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:13:27.119 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T18:13:27.342 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:13:27.554 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:13:27.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:13:28.188 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:13:28.350 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:13:28.462 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:13:28.490 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:28.490 INFO:teuthology.orchestra.run.smithi144.stdout:Last metadata expiration check: 0:02:14 ago on Fri 14 Jul 2023 06:11:14 PM UTC. 2023-07-14T18:13:28.599 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:28.609 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:13:28.763 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:13:28.881 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:28.898 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:13:28.968 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T18:13:29.048 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:13:29.141 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T18:13:29.330 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:13:29.336 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T18:13:29.467 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:13:29.616 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:13:29.625 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:13:29.783 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:13:29.942 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:13:30.138 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:13:30.216 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:13:30.405 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:13:30.418 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:13:30.420 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:13:30.572 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:13:30.580 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:13:30.583 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:13:30.716 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T18:13:30.716 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:30.942 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:30.942 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:13:31.179 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:13:31.216 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:13:31.312 INFO:teuthology.orchestra.run.smithi144.stdout:Installed products updated. 2023-07-14T18:13:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:13:31.387 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:13:31.524 INFO:teuthology.orchestra.run.smithi144.stdout:Upgraded: 2023-07-14T18:13:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:13:31.524 INFO:teuthology.orchestra.run.smithi144.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.525 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.526 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.527 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:13:31.528 INFO:teuthology.orchestra.run.smithi144.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:13:31.529 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:13:31.530 INFO:teuthology.orchestra.run.smithi144.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:13:31.531 INFO:teuthology.orchestra.run.smithi144.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:13:31.532 INFO:teuthology.orchestra.run.smithi144.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:31.533 INFO:teuthology.orchestra.run.smithi144.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.534 INFO:teuthology.orchestra.run.smithi144.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:13:31.535 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:13:31.536 INFO:teuthology.orchestra.run.smithi144.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:13:31.537 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.538 INFO:teuthology.orchestra.run.smithi144.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:13:31.539 INFO:teuthology.orchestra.run.smithi144.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:13:31.540 INFO:teuthology.orchestra.run.smithi144.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:13:31.541 INFO:teuthology.orchestra.run.smithi144.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:13:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:13:31.542 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:13:31.542 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2023-07-14T18:13:31.698 DEBUG:teuthology.parallel:result is None 2023-07-14T18:13:31.917 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:31.917 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:13:31.917 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:13:31.917 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:13:31.917 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:13:31.918 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:13:31.919 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:13:31.920 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:13:31.920 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:13:31.920 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:13:31.920 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:13:31.921 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:13:31.921 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:13:31.921 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:13:31.921 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:13:31.922 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:13:31.922 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:13:31.922 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:13:31.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:13:31.936 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:13:31.936 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:13:31.936 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:13:31.937 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:13:31.937 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:13:31.937 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:13:31.938 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:13:31.938 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:13:31.938 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:13:31.938 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:13:31.939 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:13:31.939 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:13:31.939 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:13:31.939 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:13:31.940 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:13:31.940 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:13:31.940 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:13:31.941 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:13:31.941 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:13:31.941 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:13:31.941 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:13:31.942 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:13:31.942 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:13:31.942 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:13:31.943 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:13:31.943 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:13:31.943 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:13:31.943 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:13:31.944 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:13:31.944 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:13:31.944 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:13:31.945 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:13:31.945 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:13:31.945 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:13:31.945 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:13:31.946 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:13:31.946 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:13:31.946 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:13:31.947 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:13:31.947 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:13:31.947 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T18:13:31.947 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T18:13:31.948 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T18:13:31.948 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T18:13:31.948 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T18:13:31.949 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T18:13:31.949 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T18:13:31.949 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T18:13:31.949 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T18:13:31.950 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T18:13:31.950 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T18:13:31.950 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T18:13:31.950 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T18:13:31.951 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T18:13:31.951 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T18:13:31.951 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T18:13:31.952 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T18:13:31.952 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T18:13:31.952 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T18:13:31.952 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T18:13:31.953 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T18:13:31.953 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T18:13:31.953 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T18:13:31.954 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T18:13:31.954 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T18:13:31.954 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T18:13:31.954 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T18:13:31.955 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T18:13:31.955 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T18:13:31.955 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T18:13:31.955 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T18:13:31.956 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T18:13:31.956 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T18:13:31.956 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T18:13:31.957 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T18:13:31.957 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T18:13:31.957 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T18:13:31.957 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T18:13:31.958 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T18:13:31.958 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T18:13:31.958 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T18:13:31.959 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T18:13:31.959 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T18:13:31.959 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T18:13:31.959 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T18:13:31.960 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:13:31.961 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:13:31.962 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:13:31.962 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:13:31.962 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:13:32.139 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:13:32.374 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T18:13:32.552 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T18:13:32.780 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:13:32.965 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:13:33.096 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:13:33.326 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:13:33.508 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:13:34.300 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:13:34.300 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:02:19 ago on Fri 14 Jul 2023 06:11:15 PM UTC. 2023-07-14T18:13:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:13:36.457 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T18:13:36.755 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T18:13:37.005 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T18:13:37.216 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T18:13:37.454 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:13:37.654 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:13:37.839 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:13:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:13:38.140 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T18:13:38.317 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T18:13:38.507 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:13:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:13:38.813 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T18:13:39.021 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T18:13:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:13:39.276 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:13:39.395 INFO:teuthology.orchestra.run.smithi105.stdout:Installed products updated. 2023-07-14T18:13:39.481 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T18:13:39.607 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:13:39.608 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2023-07-14T18:13:39.608 INFO:teuthology.orchestra.run.smithi105.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:13:39.609 INFO:teuthology.orchestra.run.smithi105.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.610 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.611 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.612 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:13:39.613 INFO:teuthology.orchestra.run.smithi105.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:13:39.614 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.615 INFO:teuthology.orchestra.run.smithi105.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:13:39.616 INFO:teuthology.orchestra.run.smithi105.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:13:39.616 INFO:teuthology.orchestra.run.smithi105.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:13:39.616 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:13:39.617 INFO:teuthology.orchestra.run.smithi105.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:13:39.617 INFO:teuthology.orchestra.run.smithi105.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:13:39.617 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:13:39.617 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:13:39.617 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:13:39.618 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:39.619 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:39.619 INFO:teuthology.orchestra.run.smithi105.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:13:39.619 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.620 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:13:39.621 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:13:39.622 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:13:39.623 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:13:39.624 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:13:39.625 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:13:39.626 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:13:39.627 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:13:39.628 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:13:39.628 INFO:teuthology.orchestra.run.smithi105.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:13:39.628 INFO:teuthology.orchestra.run.smithi105.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:13:39.628 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:13:39.629 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T18:13:39.629 INFO:teuthology.orchestra.run.smithi105.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:13:39.629 INFO:teuthology.orchestra.run.smithi105.stdout: 2023-07-14T18:13:39.629 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2023-07-14T18:13:39.799 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T18:13:39.809 DEBUG:teuthology.parallel:result is None 2023-07-14T18:13:40.001 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T18:13:40.169 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T18:13:40.313 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T18:13:40.543 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T18:13:40.742 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T18:13:41.143 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T18:13:41.444 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T18:13:41.662 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T18:13:41.927 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T18:13:42.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T18:13:42.467 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T18:13:42.682 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T18:13:42.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T18:13:43.098 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T18:13:43.263 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T18:13:43.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T18:13:43.791 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T18:13:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T18:13:44.307 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T18:13:44.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T18:13:44.707 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:13:44.895 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:13:45.140 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:13:45.415 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:13:45.561 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:13:45.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:13:45.938 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:13:46.139 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:13:46.370 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:13:46.552 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:13:46.705 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:13:46.925 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:13:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:13:47.355 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:13:47.602 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:13:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:13:47.965 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:13:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:13:49.364 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:13:49.637 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:13:49.900 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:13:50.208 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:13:50.482 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:13:50.691 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:13:51.609 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:13:51.789 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:13:51.864 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:13:52.031 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:13:55.850 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:13:56.439 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:13:57.075 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:13:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:13:57.401 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:13:57.538 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:14:16.971 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:14:16.972 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2023-07-14T18:14:16.972 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2023-07-14T18:14:16.972 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /dev 2023-07-14T18:14:16.972 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /run 2023-07-14T18:14:16.972 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2023-07-14T18:14:16.973 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2023-07-14T18:14:16.973 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2023-07-14T18:14:16.973 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2023-07-14T18:14:16.973 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /dev 2023-07-14T18:14:16.973 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:14:17.312 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:14:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:14:19.167 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:14:19.398 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:14:19.571 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:14:20.052 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:14:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:14:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:14:21.974 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:14:22.144 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:14:22.259 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:14:22.487 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:14:22.660 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:14:22.882 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:14:23.124 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:14:24.147 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:14:24.277 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:14:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:14:24.736 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:14:24.949 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:14:24.986 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:14:25.152 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:14:25.164 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:14:26.542 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:14:26.723 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:14:26.758 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:14:26.936 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:14:26.979 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:14:27.127 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:14:27.500 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:14:27.644 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:14:32.648 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:14:33.550 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:14:33.863 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T18:14:34.188 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T18:14:34.438 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T18:14:34.592 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:14:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:14:35.105 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T18:14:35.264 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:14:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:14:37.684 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:14:37.918 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:14:37.935 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:14:38.085 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:14:38.376 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:14:38.676 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:14:39.026 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:14:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:14:39.288 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:14:39.517 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:14:39.529 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:14:39.684 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:14:39.692 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:14:39.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T18:14:39.811 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:14:40.004 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:14:40.004 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:14:40.233 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:14:40.450 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:14:40.975 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:14:40.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:14:40.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:14:40.975 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:14:40.976 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:14:40.977 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:14:40.978 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:14:40.985 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:14:40.986 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:14:40.987 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:14:40.988 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:14:40.989 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:14:40.990 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:14:40.991 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T18:14:40.992 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T18:14:40.993 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T18:14:40.994 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T18:14:40.995 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T18:14:40.996 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T18:14:40.997 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T18:14:40.998 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:14:40.999 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:14:41.000 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:14:43.503 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:14:43.504 INFO:teuthology.orchestra.run.smithi064.stdout:Last metadata expiration check: 0:03:06 ago on Fri 14 Jul 2023 06:11:37 PM UTC. 2023-07-14T18:14:47.051 INFO:teuthology.orchestra.run.smithi064.stdout:Installed products updated. 2023-07-14T18:14:47.289 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:14:47.289 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:14:47.290 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.291 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.292 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.293 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:14:47.294 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:14:47.295 INFO:teuthology.orchestra.run.smithi064.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:14:47.296 INFO:teuthology.orchestra.run.smithi064.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:14:47.297 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:14:47.298 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.299 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:14:47.300 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:14:47.301 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:14:47.302 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.303 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:14:47.304 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:14:47.305 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:14:47.306 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:14:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:14:47.307 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:14:47.307 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2023-07-14T18:14:47.486 DEBUG:teuthology.parallel:result is None 2023-07-14T18:14:47.487 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:51.237 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:14:51.265 INFO:teuthology.orchestra.run.smithi064.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.266 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.266 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:14:51.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:51.489 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:14:51.518 INFO:teuthology.orchestra.run.smithi105.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.519 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.519 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:14:51.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:51.822 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:14:51.851 INFO:teuthology.orchestra.run.smithi144.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.852 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:14:51.852 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:14:51.852 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T18:14:51.853 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:14:51.853 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:14:51.902 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:14:51.902 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:14:51.949 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:14:51.951 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:14:51.998 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T18:14:51.999 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:14:51.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:14:52.033 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:14:52.105 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:14:52.105 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:14:52.136 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:14:52.205 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:14:52.205 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:14:52.239 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:14:52.307 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T18:14:52.308 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:14:52.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:14:52.341 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:14:52.414 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:14:52.414 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:14:52.445 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:14:52.516 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:14:52.517 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:14:52.546 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:14:52.614 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T18:14:52.615 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:14:52.615 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:14:52.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:14:52.719 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:14:52.719 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:14:52.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:14:52.821 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:14:52.822 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:14:52.850 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:14:52.919 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T18:14:53.014 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T18:14:53.015 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:53.028 INFO:tasks.cephadm:Cluster fsid is 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:14:53.029 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T18:14:53.030 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.64', 'mon.b': '172.21.15.105', 'mon.c': '172.21.15.144'} 2023-07-14T18:14:53.030 INFO:tasks.cephadm:First mon is mon.a on smithi064 2023-07-14T18:14:53.031 INFO:tasks.cephadm:First mgr is x 2023-07-14T18:14:53.031 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T18:14:53.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2023-07-14T18:14:53.068 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2023-07-14T18:14:53.101 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname $(hostname -s) 2023-07-14T18:14:53.131 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T18:14:53.132 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:53.498 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T18:14:53.778 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:14:53.779 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T18:14:53.779 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:54.918 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:14 /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:54.918 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:56.250 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:14 /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:56.250 DEBUG:teuthology.orchestra.run.smithi144:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:57.934 INFO:teuthology.orchestra.run.smithi144.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:14 /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:57.934 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:57.957 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:57.978 DEBUG:teuthology.orchestra.run.smithi144:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:14:58.013 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T18:14:58.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:14:58.016 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:14:58.020 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:14:59.377 INFO:teuthology.orchestra.run.smithi144.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:14:59.539 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:15:02.547 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:15:44.059 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:15:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:15:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:15:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2023-07-14T18:15:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:15:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2023-07-14T18:15:44.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:15:56.607 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2023-07-14T18:15:56.608 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:15:56.608 INFO:teuthology.orchestra.run.smithi144.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:15:56.608 INFO:teuthology.orchestra.run.smithi144.stdout: "repo_digests": [ 2023-07-14T18:15:56.608 INFO:teuthology.orchestra.run.smithi144.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:15:56.608 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2023-07-14T18:15:56.609 INFO:teuthology.orchestra.run.smithi144.stdout:} 2023-07-14T18:15:56.771 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2023-07-14T18:15:56.771 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:15:56.772 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:15:56.772 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2023-07-14T18:15:56.772 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:15:56.772 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2023-07-14T18:15:56.772 INFO:teuthology.orchestra.run.smithi105.stdout:} 2023-07-14T18:15:56.790 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2023-07-14T18:15:56.829 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2023-07-14T18:15:56.858 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph 2023-07-14T18:15:56.889 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2023-07-14T18:15:56.921 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2023-07-14T18:15:56.947 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /etc/ceph 2023-07-14T18:15:56.973 INFO:tasks.cephadm:Writing seed config... 2023-07-14T18:15:56.974 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T18:15:56.975 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T18:15:56.975 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T18:15:56.975 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T18:15:56.975 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T18:15:56.975 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T18:15:56.976 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T18:15:56.976 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T18:15:56.976 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T18:15:56.976 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T18:15:56.976 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T18:15:56.977 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T18:15:56.977 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T18:15:56.977 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T18:15:56.977 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T18:15:56.977 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T18:15:56.978 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T18:15:56.978 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T18:15:56.978 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T18:15:56.978 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T18:15:56.978 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T18:15:56.979 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T18:15:56.980 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T18:15:56.980 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T18:15:56.980 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-14T18:15:56.980 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-14T18:15:56.980 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T18:15:56.981 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T18:15:56.981 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-14T18:15:56.981 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2023-07-14T18:15:56.981 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T18:15:56.981 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T18:15:56.982 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T18:15:56.982 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T18:15:56.982 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2023-07-14T18:15:56.982 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T18:15:56.982 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T18:15:56.983 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T18:15:56.983 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-14T18:15:56.984 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:15:56.984 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T18:15:57.003 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 5441c8fe-2272-11ee-9b33-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T18:15:57.004 DEBUG:teuthology.orchestra.run.smithi064:mon.a> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T18:15:57.046 DEBUG:teuthology.orchestra.run.smithi064:mgr.x> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T18:15:57.077 INFO:journalctl@ceph.mon.a.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:15:57.078 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T18:15:57.079 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.64 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:15:57.107 INFO:journalctl@ceph.mgr.x.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:15:57.250 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:15:57.251 INFO:teuthology.orchestra.run.smithi064.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', '5441c8fe-2272-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.64', '--skip-admin-label'] 2023-07-14T18:15:57.399 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:15:57.399 INFO:teuthology.orchestra.run.smithi064.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T18:15:57.400 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying podman|docker is present... 2023-07-14T18:15:57.544 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:15:57.544 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying lvm2 is present... 2023-07-14T18:15:57.544 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying time synchronization is in place... 2023-07-14T18:15:57.551 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:15:57.551 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:15:57.557 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:15:57.557 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:15:57.563 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2023-07-14T18:15:57.569 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2023-07-14T18:15:57.569 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:15:57.569 INFO:teuthology.orchestra.run.smithi064.stdout:Repeating the final host check... 2023-07-14T18:15:57.715 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:15:57.715 INFO:teuthology.orchestra.run.smithi064.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T18:15:57.715 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl is present 2023-07-14T18:15:57.715 INFO:teuthology.orchestra.run.smithi064.stdout:lvcreate is present 2023-07-14T18:15:57.721 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:15:57.721 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:15:57.727 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:15:57.727 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:15:57.733 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout enabled 2023-07-14T18:15:57.739 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout active 2023-07-14T18:15:57.740 INFO:teuthology.orchestra.run.smithi064.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:15:57.740 INFO:teuthology.orchestra.run.smithi064.stdout:Host looks OK 2023-07-14T18:15:57.740 INFO:teuthology.orchestra.run.smithi064.stdout:Cluster fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:15:57.741 INFO:teuthology.orchestra.run.smithi064.stdout:Acquiring lock 140688201959912 on /run/cephadm/5441c8fe-2272-11ee-9b33-001a4aab830c.lock 2023-07-14T18:15:57.741 INFO:teuthology.orchestra.run.smithi064.stdout:Lock 140688201959912 acquired on /run/cephadm/5441c8fe-2272-11ee-9b33-001a4aab830c.lock 2023-07-14T18:15:57.741 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 3300 ... 2023-07-14T18:15:57.742 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying IP 172.21.15.64 port 6789 ... 2023-07-14T18:15:57.742 INFO:teuthology.orchestra.run.smithi064.stdout:Base mon IP(s) is [172.21.15.64:3300, 172.21.15.64:6789], mon addrv is [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2023-07-14T18:15:57.744 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-14T18:15:57.744 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.64 metric 100 2023-07-14T18:15:57.746 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T18:15:57.747 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-14T18:15:57.747 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-14T18:15:57.749 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T18:15:57.749 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T18:15:57.749 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:15:57.749 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-14T18:15:57.750 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout inet6 fe80::bf40:81a3:b7e0:f244/64 scope link noprefixroute 2023-07-14T18:15:57.750 INFO:teuthology.orchestra.run.smithi064.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:15:57.750 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2023-07-14T18:15:57.750 INFO:teuthology.orchestra.run.smithi064.stdout:Mon IP `172.21.15.64` is in CIDR network `172.21.0.0/20` 2023-07-14T18:15:57.751 INFO:teuthology.orchestra.run.smithi064.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T18:15:57.751 INFO:teuthology.orchestra.run.smithi064.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T18:15:57.752 INFO:teuthology.orchestra.run.smithi064.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:15:58.195 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:15:58.195 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T18:15:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:/bin/podman: stderr Storing signatures 2023-07-14T18:15:58.952 INFO:teuthology.orchestra.run.smithi064.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:15:58.952 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:15:58.952 INFO:teuthology.orchestra.run.smithi064.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T18:15:59.601 INFO:teuthology.orchestra.run.smithi064.stdout:stat: stdout 167 167 2023-07-14T18:15:59.601 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial keys... 2023-07-14T18:16:00.254 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDfkLFkRT2JMxAAJsTJe5wRxHlRwnsw3u04sw== 2023-07-14T18:16:00.903 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDgkLFkIW9bHxAAEdwTM/uCPkuxT44bCJVGQg== 2023-07-14T18:16:01.553 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph-authtool: stdout AQDhkLFkafUDCxAAQzUvebOhptgDioZZ7pqYuA== 2023-07-14T18:16:01.553 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial monmap... 2023-07-14T18:16:02.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:16:02.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T18:16:02.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:02.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:16:02.214 INFO:teuthology.orchestra.run.smithi064.stdout:monmaptool for a [v2:172.21.15.64:3300,v1:172.21.15.64:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:16:02.214 INFO:teuthology.orchestra.run.smithi064.stdout:setting min_mon_release = pacific 2023-07-14T18:16:02.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: set fsid to 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:02.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:16:02.215 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:16:02.215 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mon... 2023-07-14T18:16:02.959 INFO:teuthology.orchestra.run.smithi064.stdout:create mon.a on 2023-07-14T18:16:03.089 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T18:16:03.179 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T18:16:03.305 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target. 2023-07-14T18:16:03.305 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target. 2023-07-14T18:16:03.493 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a 2023-07-14T18:16:03.494 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T18:16:03.586 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target.wants/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:16:03.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:03 smithi064 systemd[1]: Starting Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:16:04.140 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:03 smithi064 podman[96172]: 2023-07-14T18:16:04.141 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:04 smithi064 bash[96172]: 1d7e91ea10c11a3a58294a1dd96b866bc50e624c98edaa7b002de05a571edaa3 2023-07-14T18:16:04.196 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:16:04.196 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2023-07-14T18:16:04.204 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:16:04.205 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:16:04.205 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2023-07-14T18:16:04.205 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:16:04.205 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon to start... 2023-07-14T18:16:04.206 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mon... 2023-07-14T18:16:04.420 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:04 smithi064 ceph-mon[96212]: mkfs 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:04.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:04 smithi064 ceph-mon[96212]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:16:04.421 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:04 smithi064 systemd[1]: Started Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:16:05.155 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T18:16:05.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout id: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:05.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T18:16:05.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:05.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout services: 2023-07-14T18:16:05.156 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.690802s) 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout data: 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T18:16:05.157 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.643+0000 7fc243a6f700 1 Processor -- start 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc243a6f700 1 -- start start 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc243a6f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc243a6f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c073be0 con 0x7fc23c105df0 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc24180b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:05.158 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc24180b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45506/0 (socket says 172.21.15.64:45506) 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.644+0000 7fc24180b700 1 -- 172.21.15.64:0/3366727347 learned_addr learned my addr 172.21.15.64:0/3366727347 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc24180b700 1 -- 172.21.15.64:0/3366727347 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c073d20 con 0x7fc23c105df0 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc24180b700 1 --2- 172.21.15.64:0/3366727347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098a0 tx=0x7fc238009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297503b75cbaa75e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc240809700 1 -- 172.21.15.64:0/3366727347 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc238004160 con 0x7fc23c105df0 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc240809700 1 -- 172.21.15.64:0/3366727347 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2380042c0 con 0x7fc23c105df0 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc240809700 1 -- 172.21.15.64:0/3366727347 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc2380045a0 con 0x7fc23c105df0 2023-07-14T18:16:05.159 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc243a6f700 1 -- 172.21.15.64:0/3366727347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 msgr2=0x7fc23c1081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.645+0000 7fc243a6f700 1 --2- 172.21.15.64:0/3366727347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc2380098a0 tx=0x7fc238009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- 172.21.15.64:0/3366727347 shutdown_connections 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 --2- 172.21.15.64:0/3366727347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c1081d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- 172.21.15.64:0/3366727347 >> 172.21.15.64:0/3366727347 conn(0x7fc23c0ff850 msgr2=0x7fc23c101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- 172.21.15.64:0/3366727347 shutdown_connections 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- 172.21.15.64:0/3366727347 wait complete. 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 Processor -- start 2023-07-14T18:16:05.160 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- start start 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.646+0000 7fc243a6f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c105c90 con 0x7fc23c105df0 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc24180b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc24180b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45508/0 (socket says 172.21.15.64:45508) 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc24180b700 1 -- 172.21.15.64:0/990217371 learned_addr learned my addr 172.21.15.64:0/990217371 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:05.161 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc24180b700 1 -- 172.21.15.64:0/990217371 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc238009550 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc24180b700 1 --2- 172.21.15.64:0/990217371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc238004000 tx=0x7fc238004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc238011630 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc238011790 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c103da0 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc238011a30 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c1041c0 con 0x7fc23c105df0 2023-07-14T18:16:05.162 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc238022070 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.647+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc23801c450 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.648+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c104bd0 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.650+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fc23801b450 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.782+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc23c192240 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.783+0000 7fc22effd700 1 -- 172.21.15.64:0/990217371 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fc23801b5f0 con 0x7fc23c105df0 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 msgr2=0x7fc23c105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 --2- 172.21.15.64:0/990217371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc238004000 tx=0x7fc238004750 comp rx=0 tx=0).stop 2023-07-14T18:16:05.163 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 shutdown_connections 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 --2- 172.21.15.64:0/990217371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc23c105df0 0x7fc23c105750 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 >> 172.21.15.64:0/990217371 conn(0x7fc23c0ff850 msgr2=0x7fc23c101270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 shutdown_connections 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:04.784+0000 7fc243a6f700 1 -- 172.21.15.64:0/990217371 wait complete. 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:mon is available 2023-07-14T18:16:05.164 INFO:teuthology.orchestra.run.smithi064.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T18:16:05.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:16:05.407 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: monmap e1: 1 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0]} removed_ranks: {} 2023-07-14T18:16:05.408 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: fsmap 2023-07-14T18:16:05.408 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:16:05.408 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: mgrmap e1: no daemons active 2023-07-14T18:16:05.408 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:05 smithi064 ceph-mon[96212]: from='client.? 172.21.15.64:0/990217371' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T18:16:06.101 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.64:3300,v1:172.21.15.64:6789] 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:16:06.102 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:06.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:16:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-14T18:16:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:16:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:16:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:16:06.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.585+0000 7fb2a0102700 1 Processor -- start 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.585+0000 7fb2a0102700 1 -- start start 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb2a0102700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb2a0102700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb298073ba0 con 0x7fb298105e30 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb29de9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb29de9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45510/0 (socket says 172.21.15.64:45510) 2023-07-14T18:16:06.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb29de9e700 1 -- 172.21.15.64:0/925579266 learned_addr learned my addr 172.21.15.64:0/925579266 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.586+0000 7fb29de9e700 1 -- 172.21.15.64:0/925579266 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298073ce0 con 0x7fb298105e30 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb29de9e700 1 --2- 172.21.15.64:0/925579266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb2940098a0 tx=0x7fb294009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6d7c564846f6d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb29ce9c700 1 -- 172.21.15.64:0/925579266 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb294004160 con 0x7fb298105e30 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb29ce9c700 1 -- 172.21.15.64:0/925579266 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2940042c0 con 0x7fb298105e30 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb29ce9c700 1 -- 172.21.15.64:0/925579266 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb2940045a0 con 0x7fb298105e30 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 -- 172.21.15.64:0/925579266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 msgr2=0x7fb298108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 --2- 172.21.15.64:0/925579266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb2940098a0 tx=0x7fb294009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:06.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 -- 172.21.15.64:0/925579266 shutdown_connections 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 --2- 172.21.15.64:0/925579266 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298108210 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 -- 172.21.15.64:0/925579266 >> 172.21.15.64:0/925579266 conn(0x7fb2980ff850 msgr2=0x7fb298101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.587+0000 7fb2a0102700 1 -- 172.21.15.64:0/925579266 shutdown_connections 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.588+0000 7fb2a0102700 1 -- 172.21.15.64:0/925579266 wait complete. 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.588+0000 7fb2a0102700 1 Processor -- start 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.588+0000 7fb2a0102700 1 -- start start 2023-07-14T18:16:06.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb2a0102700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb2a0102700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb298105c80 con 0x7fb298105e30 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb29de9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb29de9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45512/0 (socket says 172.21.15.64:45512) 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb29de9e700 1 -- 172.21.15.64:0/1508659811 learned_addr learned my addr 172.21.15.64:0/1508659811 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb29de9e700 1 -- 172.21.15.64:0/1508659811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb294009550 con 0x7fb298105e30 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb29de9e700 1 --2- 172.21.15.64:0/1508659811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb294004000 tx=0x7fb294004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb294011630 con 0x7fb298105e30 2023-07-14T18:16:06.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294011790 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb298103de0 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb294011a30 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.589+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb298104260 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.590+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb294022070 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.590+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb29401c450 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.590+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb298061c70 con 0x7fb298105e30 2023-07-14T18:16:06.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.593+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fb29401b450 con 0x7fb298105e30 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.719+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb298192580 con 0x7fb298105e30 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.722+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fb29401b5f0 con 0x7fb298105e30 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.722+0000 7fb28affd700 1 -- 172.21.15.64:0/1508659811 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7fb294037540 con 0x7fb298105e30 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.723+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 msgr2=0x7fb298105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.723+0000 7fb2a0102700 1 --2- 172.21.15.64:0/1508659811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb294004000 tx=0x7fb294004750 comp rx=0 tx=0).stop 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.723+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 shutdown_connections 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.723+0000 7fb2a0102700 1 --2- 172.21.15.64:0/1508659811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb298105e30 0x7fb298105740 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:06.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.724+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 >> 172.21.15.64:0/1508659811 conn(0x7fb2980ff850 msgr2=0x7fb2981012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.724+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 shutdown_connections 2023-07-14T18:16:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:05.724+0000 7fb2a0102700 1 -- 172.21.15.64:0/1508659811 wait complete. 2023-07-14T18:16:06.111 INFO:teuthology.orchestra.run.smithi064.stdout:Generating new minimal ceph.conf... 2023-07-14T18:16:06.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:06 smithi064 ceph-mon[96212]: from='client.? 172.21.15.64:0/1508659811' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:16:06.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:06 smithi064 ceph-mon[96212]: from='client.? 172.21.15.64:0/1508659811' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T18:16:07.046 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feecf6700 1 Processor -- start 2023-07-14T18:16:07.046 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feecf6700 1 -- start start 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feecf6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feecf6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8073ba0 con 0x7f2fe8105e60 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feca92700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feca92700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45514/0 (socket says 172.21.15.64:45514) 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.529+0000 7f2feca92700 1 -- 172.21.15.64:0/4160046955 learned_addr learned my addr 172.21.15.64:0/4160046955 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2feca92700 1 -- 172.21.15.64:0/4160046955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8073ce0 con 0x7f2fe8105e60 2023-07-14T18:16:07.047 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2feca92700 1 --2- 172.21.15.64:0/4160046955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098a0 tx=0x7f2fdc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20896471b652d74 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2fe77fe700 1 -- 172.21.15.64:0/4160046955 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fdc004160 con 0x7f2fe8105e60 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2fe77fe700 1 -- 172.21.15.64:0/4160046955 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2fdc0042c0 con 0x7f2fe8105e60 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2fe77fe700 1 -- 172.21.15.64:0/4160046955 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fdc011940 con 0x7f2fe8105e60 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2feecf6700 1 -- 172.21.15.64:0/4160046955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 msgr2=0x7f2fe8108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.530+0000 7f2feecf6700 1 --2- 172.21.15.64:0/4160046955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0098a0 tx=0x7f2fdc009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- 172.21.15.64:0/4160046955 shutdown_connections 2023-07-14T18:16:07.048 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 --2- 172.21.15.64:0/4160046955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe8108240 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- 172.21.15.64:0/4160046955 >> 172.21.15.64:0/4160046955 conn(0x7f2fe80ff880 msgr2=0x7f2fe8101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- 172.21.15.64:0/4160046955 shutdown_connections 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- 172.21.15.64:0/4160046955 wait complete. 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 Processor -- start 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- start start 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.531+0000 7f2feecf6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8105d00 con 0x7f2fe8105e60 2023-07-14T18:16:07.049 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feca92700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feca92700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45516/0 (socket says 172.21.15.64:45516) 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feca92700 1 -- 172.21.15.64:0/1625259881 learned_addr learned my addr 172.21.15.64:0/1625259881 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feca92700 1 -- 172.21.15.64:0/1625259881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc009550 con 0x7f2fe8105e60 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feca92700 1 --2- 172.21.15.64:0/1625259881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000c00 tx=0x7f2fdc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fdc0036a0 con 0x7f2fe8105e60 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2fdc003c70 con 0x7f2fe8105e60 2023-07-14T18:16:07.050 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe8103e10 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fdc0193f0 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.532+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe8104290 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.533+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fdc019970 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.533+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2fdc01a3d0 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.534+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe8109320 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.536+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2fdc029060 con 0x7f2fe8105e60 2023-07-14T18:16:07.051 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.662+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2fe81048f0 con 0x7f2fe8105e60 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.663+0000 7f2fe5ffb700 1 -- 172.21.15.64:0/1625259881 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2fdc01f070 con 0x7f2fe8105e60 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 msgr2=0x7f2fe81057c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 --2- 172.21.15.64:0/1625259881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc000c00 tx=0x7f2fdc011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 shutdown_connections 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 --2- 172.21.15.64:0/1625259881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2fe8105e60 0x7f2fe81057c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 >> 172.21.15.64:0/1625259881 conn(0x7f2fe80ff880 msgr2=0x7f2fe8100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:07.052 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 shutdown_connections 2023-07-14T18:16:07.053 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:06.664+0000 7f2feecf6700 1 -- 172.21.15.64:0/1625259881 wait complete. 2023-07-14T18:16:07.053 INFO:teuthology.orchestra.run.smithi064.stdout:Restarting the monitor... 2023-07-14T18:16:07.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:07 smithi064 systemd[1]: Stopping Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:16:07.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:07 smithi064 ceph-mon[96212]: from='client.? 172.21.15.64:0/1625259881' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:16:07.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:07 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a[96188]: 2023-07-14T18:16:07.334+0000 7f861d080700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:16:07.680 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:07 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a[96188]: 2023-07-14T18:16:07.334+0000 7f861d080700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T18:16:08.058 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:07 smithi064 bash[96552]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a 2023-07-14T18:16:08.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 systemd[1]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T18:16:08.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 systemd[1]: Stopped Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:16:08.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 systemd[1]: Starting Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:16:08.610 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 podman[96680]: 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: pidfile_write: ignore empty --pid-file 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: load: jerasure load: lrc 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Git sha 0 2023-07-14T18:16:08.611 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: DB SUMMARY 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: DB Session ID: GE85TS7CABSUDHSGKTE6 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T18:16:08.612 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 98044 ; 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.env: 0x56218d784c60 2023-07-14T18:16:08.613 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.info_log: 0x56218f2f72a0 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.statistics: (nil) 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.use_fsync: 0 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:16:08.614 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:16:08.615 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:16:08.616 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:16:08.616 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.db_log_dir: 2023-07-14T18:16:08.616 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.wal_dir: 2023-07-14T18:16:08.617 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:16:08.617 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:16:08.617 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.write_buffer_manager: 0x56218f2183c0 2023-07-14T18:16:08.618 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:16:08.619 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.unordered_write: 0 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.row_cache: None 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.wal_filter: None 2023-07-14T18:16:08.620 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.wal_compression: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:16:08.621 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:16:08.622 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:16:08.623 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_open_files: -1 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:16:08.624 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Compression algorithms supported: 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kZSTD supported: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:16:08.625 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T18:16:08.626 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.merge_operator: 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_filter: None 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:16:08.627 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56218f2f73a0) 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_type: 0 2023-07-14T18:16:08.628 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_index_type: 0 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_shortening: 1 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: checksum: 4 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: no_block_cache: 0 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache: 0x56218f2b71f0 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_options: 2023-07-14T18:16:08.629 INFO:journalctl@ceph.mon.a.smithi064.stdout: capacity : 536870912 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_shard_bits : 4 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: strict_capacity_limit : 0 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_cache_compressed: (nil) 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: persistent_cache: (nil) 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size: 4096 2023-07-14T18:16:08.630 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_size_deviation: 10 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_restart_interval: 16 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: index_block_restart_interval: 1 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: metadata_block_size: 4096 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: partition_filters: 0 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: use_delta_encoding: 1 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: filter_policy: bloomfilter 2023-07-14T18:16:08.631 INFO:journalctl@ceph.mon.a.smithi064.stdout: whole_key_filtering: 1 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: verify_compression: 0 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: format_version: 5 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_index_compression: 1 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: block_align: 0 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: max_auto_readahead_size: 262144 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: prepopulate_block_cache: 0 2023-07-14T18:16:08.632 INFO:journalctl@ceph.mon.a.smithi064.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression: NoCompression 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:16:08.633 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.num_levels: 7 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:16:08.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:16:08.635 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:16:08.636 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:16:08.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:16:08.638 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:16:08.639 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:16:08.640 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:16:08.641 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.ttl: 2592000 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:16:08.642 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:16:08.643 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e32dcec2-eedc-4ec8-8c61-0a0a8694ed83 2023-07-14T18:16:08.644 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358568489572, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358568490651, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93381, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 312, "table_properties": {"data_size": 91381, "index_size": 282, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13491, "raw_average_key_size": 49, "raw_value_size": 84001, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 271, "num_filter_entries": 271, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689358568, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e32dcec2-eedc-4ec8-8c61-0a0a8694ed83", "db_session_id": "GE85TS7CABSUDHSGKTE6", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358568490723, "job": 1, "event": "recovery_finished"} 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56218f350000 2023-07-14T18:16:08.645 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: DB pointer 0x56218f33c000 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** DB Stats ** 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:16:08.646 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:16:08.647 INFO:journalctl@ceph.mon.a.smithi064.stdout: L0 2/0 93.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: Sum 2/0 93.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 132.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** Compaction Stats [default] ** 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 132.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:16:08.648 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:16:08.649 INFO:journalctl@ceph.mon.a.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: Cumulative compaction: 0.00 GB write, 15.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: Interval compaction: 0.00 GB write, 15.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache BinnedLRUCache@0x56218f2b71f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:16:08.650 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: starting mon.a rank 0 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???) e1 preinit fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).mds e1 new map 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).mds e1 print_map 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout: e1 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:16:08.651 INFO:journalctl@ceph.mon.a.smithi064.stdout: legacy client fscid: -1 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout: 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout: No filesystems configured 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:16:08.652 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expand map: {default=false} 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expand map: {default=info} 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:16:08.653 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 bash[96680]: e874a9b4921ec69e5bbc632e173d70a59880d58a7932d0922c46e4b4536c78ba 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: monmap e1: 1 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0]} removed_ranks: {} 2023-07-14T18:16:08.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: fsmap 2023-07-14T18:16:08.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:16:08.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 ceph-mon[96720]: mgrmap e1: no daemons active 2023-07-14T18:16:08.655 INFO:teuthology.orchestra.run.smithi064.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T18:16:08.879 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:08 smithi064 systemd[1]: Started Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:16:10.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:10 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3861788157' entity='client.admin' 2023-07-14T18:16:10.998 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.346+0000 7fe181a3f700 1 Processor -- start 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.346+0000 7fe181a3f700 1 -- start start 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.346+0000 7fe181a3f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.346+0000 7fe181a3f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c103e80 con 0x7fe17c103530 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe17affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe17affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45518/0 (socket says 172.21.15.64:45518) 2023-07-14T18:16:10.999 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe17affd700 1 -- 172.21.15.64:0/2749892518 learned_addr learned my addr 172.21.15.64:0/2749892518 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe17affd700 1 -- 172.21.15.64:0/2749892518 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c103fc0 con 0x7fe17c103530 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe17affd700 1 --2- 172.21.15.64:0/2749892518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0098a0 tx=0x7fe16c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c797d503e3e7fc61 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe179ffb700 1 -- 172.21.15.64:0/2749892518 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe16c004160 con 0x7fe17c103530 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.347+0000 7fe179ffb700 1 -- 172.21.15.64:0/2749892518 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe16c0042c0 con 0x7fe17c103530 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe179ffb700 1 -- 172.21.15.64:0/2749892518 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe16c011940 con 0x7fe17c103530 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 -- 172.21.15.64:0/2749892518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 msgr2=0x7fe17c103940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:11.000 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 --2- 172.21.15.64:0/2749892518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe16c0098a0 tx=0x7fe16c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 -- 172.21.15.64:0/2749892518 shutdown_connections 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 --2- 172.21.15.64:0/2749892518 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c103940 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 -- 172.21.15.64:0/2749892518 >> 172.21.15.64:0/2749892518 conn(0x7fe17c0fef30 msgr2=0x7fe17c101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 -- 172.21.15.64:0/2749892518 shutdown_connections 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.348+0000 7fe181a3f700 1 -- 172.21.15.64:0/2749892518 wait complete. 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe181a3f700 1 Processor -- start 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe181a3f700 1 -- start start 2023-07-14T18:16:11.001 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe181a3f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe181a3f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c1993c0 con 0x7fe17c103530 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe17affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe17affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45520/0 (socket says 172.21.15.64:45520) 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe17affd700 1 -- 172.21.15.64:0/3861788157 learned_addr learned my addr 172.21.15.64:0/3861788157 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.349+0000 7fe17affd700 1 -- 172.21.15.64:0/3861788157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe16c009550 con 0x7fe17c103530 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe17affd700 1 --2- 172.21.15.64:0/3861788157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000c00 tx=0x7fe16c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe16c0036a0 con 0x7fe17c103530 2023-07-14T18:16:11.002 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17c1995c0 con 0x7fe17c103530 2023-07-14T18:16:11.004 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe16c01b440 con 0x7fe17c103530 2023-07-14T18:16:11.004 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe16c0193f0 con 0x7fe17c103530 2023-07-14T18:16:11.004 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17c199a60 con 0x7fe17c103530 2023-07-14T18:16:11.004 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe16c019930 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.350+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe16c01a3d0 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.351+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17c061c70 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.353+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe16c029060 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.479+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe17c107190 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe180a3d700 1 -- 172.21.15.64:0/3861788157 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe16c01f070 con 0x7fe17c103530 2023-07-14T18:16:11.005 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 msgr2=0x7fe17c198e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe181a3f700 1 --2- 172.21.15.64:0/3861788157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000c00 tx=0x7fe16c011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 shutdown_connections 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe181a3f700 1 --2- 172.21.15.64:0/3861788157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe17c103530 0x7fe17c198e80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.482+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 >> 172.21.15.64:0/3861788157 conn(0x7fe17c0fef30 msgr2=0x7fe17c0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.483+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 shutdown_connections 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:09.483+0000 7fe181a3f700 1 -- 172.21.15.64:0/3861788157 wait complete. 2023-07-14T18:16:11.006 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T18:16:11.105 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:16:11.106 INFO:teuthology.orchestra.run.smithi064.stdout:Creating mgr... 2023-07-14T18:16:11.106 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 9283 ... 2023-07-14T18:16:11.106 INFO:teuthology.orchestra.run.smithi064.stdout:Verifying port 8765 ... 2023-07-14T18:16:11.511 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T18:16:11.512 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Failed to reset failed state of unit ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T18:16:11.612 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c.target.wants/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:16:11.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:11 smithi064 systemd[1]: Starting Ceph mgr.x for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:16:13.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:12 smithi064 podman[96952]: 2023-07-14T18:16:14.627 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:16:14.628 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2023-07-14T18:16:14.635 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:16:14.635 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:16:14.635 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2023-07-14T18:16:14.635 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:16:14.643 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:16:14.644 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2023-07-14T18:16:14.651 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:16:14.652 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:16:14.652 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2023-07-14T18:16:14.652 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T18:16:14.652 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr to start... 2023-07-14T18:16:14.653 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr... 2023-07-14T18:16:14.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 bash[96952]: c9d8d66dde859594ff98d296adbe7b7ca8947fdc260e3a6ace91369cbf9763ea 2023-07-14T18:16:14.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 systemd[1]: Started Ceph mgr.x for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:16:14.836 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:14.581+0000 7f32dda81200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:16:14.836 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:14.694+0000 7f32dda81200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:16:15.169 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:14.803+0000 7f32dda81200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:16:15.170 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:14 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:14.913+0000 7f32dda81200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:16:15.824 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:15 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:15.121+0000 7f32dda81200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:16:15.824 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:15 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:15.520+0000 7f32dda81200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:16:15.825 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:15 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:15.628+0000 7f32dda81200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:16:16.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:15 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:15.779+0000 7f32dda81200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:16:16.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:15 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:15.894+0000 7f32dda81200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:16:16.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:16 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:16.417+0000 7f32dda81200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:16:17.428 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:17 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3424557065' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:16:17.428 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:17 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:17.128+0000 7f32dda81200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:16:17.429 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:17 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:17.326+0000 7f32dda81200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:16:17.616 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:16:17.617 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:16:17.618 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.619 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:16:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:16:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:16:17.620 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:16:17.625 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:16:17.626 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:16:17.627 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:17.628 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:16:17.628 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:16:17.628 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.628 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:16:17.628 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:17.629 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:16:04.094856+0000", 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:17.630 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.048+0000 7fe74f361700 1 Processor -- start 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.048+0000 7fe74f361700 1 -- start start 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.048+0000 7fe74f361700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.048+0000 7fe74f361700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748104b30 con 0x7fe7481041e0 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.048+0000 7fe74d0fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe74d0fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45526/0 (socket says 172.21.15.64:45526) 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe74d0fd700 1 -- 172.21.15.64:0/1008621567 learned_addr learned my addr 172.21.15.64:0/1008621567 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:17.631 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe74d0fd700 1 -- 172.21.15.64:0/1008621567 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748104c70 con 0x7fe7481041e0 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe74d0fd700 1 --2- 172.21.15.64:0/1008621567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe74400bb40 tx=0x7fe74400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350e75d43c7a86a6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe73ffff700 1 -- 172.21.15.64:0/1008621567 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe74400dbe0 con 0x7fe7481041e0 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe73ffff700 1 -- 172.21.15.64:0/1008621567 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe7440044a0 con 0x7fe7481041e0 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.049+0000 7fe73ffff700 1 -- 172.21.15.64:0/1008621567 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe744003da0 con 0x7fe7481041e0 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 -- 172.21.15.64:0/1008621567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 msgr2=0x7fe7481045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 --2- 172.21.15.64:0/1008621567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe74400bb40 tx=0x7fe74400be50 comp rx=0 tx=0).stop 2023-07-14T18:16:17.632 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 -- 172.21.15.64:0/1008621567 shutdown_connections 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 --2- 172.21.15.64:0/1008621567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe7481045f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 -- 172.21.15.64:0/1008621567 >> 172.21.15.64:0/1008621567 conn(0x7fe7480ff850 msgr2=0x7fe748101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 -- 172.21.15.64:0/1008621567 shutdown_connections 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.050+0000 7fe74f361700 1 -- 172.21.15.64:0/1008621567 wait complete. 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74f361700 1 Processor -- start 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74f361700 1 -- start start 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74f361700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:17.633 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74f361700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe748197340 con 0x7fe7481041e0 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74d0fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74d0fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45528/0 (socket says 172.21.15.64:45528) 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.051+0000 7fe74d0fd700 1 -- 172.21.15.64:0/3424557065 learned_addr learned my addr 172.21.15.64:0/3424557065 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe74d0fd700 1 -- 172.21.15.64:0/3424557065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74400b7f0 con 0x7fe7481041e0 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe74d0fd700 1 --2- 172.21.15.64:0/3424557065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe74400e010 tx=0x7fe744003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe74401e070 con 0x7fe7481041e0 2023-07-14T18:16:17.634 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fe744010970 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe744022a10 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe748197540 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe748197960 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe74402c740 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.052+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe744019a60 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.053+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe748190b10 con 0x7fe7481041e0 2023-07-14T18:16:17.635 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.055+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fe744027070 con 0x7fe7481041e0 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.189+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe748106140 con 0x7fe7481041e0 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.189+0000 7fe73e7fc700 1 -- 172.21.15.64:0/3424557065 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fe744010e10 con 0x7fe7481041e0 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 msgr2=0x7fe748196e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 --2- 172.21.15.64:0/3424557065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe74400e010 tx=0x7fe744003f70 comp rx=0 tx=0).stop 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 shutdown_connections 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 --2- 172.21.15.64:0/3424557065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe7481041e0 0x7fe748196e00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 >> 172.21.15.64:0/3424557065 conn(0x7fe7480ff850 msgr2=0x7fe74818dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:17.636 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 shutdown_connections 2023-07-14T18:16:17.637 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:17.190+0000 7fe74f361700 1 -- 172.21.15.64:0/3424557065 wait complete. 2023-07-14T18:16:17.637 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (1/15)... 2023-07-14T18:16:18.670 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:18 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:18.296+0000 7f32dda81200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:16:18.670 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:18 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:18.406+0000 7f32dda81200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:16:19.016 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:18 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:18.624+0000 7f32dda81200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:16:19.271 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:18 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:18.971+0000 7f32dda81200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:16:19.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:19 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:19.225+0000 7f32dda81200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:16:19.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:19 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:19.407+0000 7f32dda81200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:16:20.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:19 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:19.738+0000 7f32dda81200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:16:20.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1692063256' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:16:20.667 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:20.668 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:20.668 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:16:20.668 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:16:20.668 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:16:20.668 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:20.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:16:20.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:16:20.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:20.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:16:20.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:16:20.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:16:20.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:16:20.677 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.678 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:16:20.679 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:16:20.680 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:16:04.094856+0000", 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f715af0c700 1 Processor -- start 2023-07-14T18:16:20.681 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f715af0c700 1 -- start start 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f715af0c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f715af0c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154105c40 con 0x7f71541052f0 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f7158ca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f7158ca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45530/0 (socket says 172.21.15.64:45530) 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f7158ca8700 1 -- 172.21.15.64:0/1105644939 learned_addr learned my addr 172.21.15.64:0/1105644939 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.120+0000 7f7158ca8700 1 -- 172.21.15.64:0/1105644939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7154105d80 con 0x7f71541052f0 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f7158ca8700 1 --2- 172.21.15.64:0/1105644939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f71480098a0 tx=0x7f7148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ffa6525005b445 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:20.682 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f71537fe700 1 -- 172.21.15.64:0/1105644939 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7148004160 con 0x7f71541052f0 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f71537fe700 1 -- 172.21.15.64:0/1105644939 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f71480042c0 con 0x7f71541052f0 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f71537fe700 1 -- 172.21.15.64:0/1105644939 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7148011940 con 0x7f71541052f0 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f715af0c700 1 -- 172.21.15.64:0/1105644939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 msgr2=0x7f7154105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.121+0000 7f715af0c700 1 --2- 172.21.15.64:0/1105644939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f71480098a0 tx=0x7f7148009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- 172.21.15.64:0/1105644939 shutdown_connections 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 --2- 172.21.15.64:0/1105644939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154105700 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:20.683 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- 172.21.15.64:0/1105644939 >> 172.21.15.64:0/1105644939 conn(0x7f7154100960 msgr2=0x7f7154102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- 172.21.15.64:0/1105644939 shutdown_connections 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- 172.21.15.64:0/1105644939 wait complete. 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 Processor -- start 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- start start 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.122+0000 7f715af0c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7154199560 con 0x7f71541052f0 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7158ca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:20.684 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7158ca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45532/0 (socket says 172.21.15.64:45532) 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7158ca8700 1 -- 172.21.15.64:0/1692063256 learned_addr learned my addr 172.21.15.64:0/1692063256 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7158ca8700 1 -- 172.21.15.64:0/1692063256 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148009550 con 0x7f71541052f0 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7158ca8700 1 --2- 172.21.15.64:0/1692063256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7148011dc0 tx=0x7f7148011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71480037d0 con 0x7f71541052f0 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f714801b440 con 0x7f71541052f0 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7154199760 con 0x7f71541052f0 2023-07-14T18:16:20.685 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.123+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f714801a560 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.124+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7154199b80 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.124+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f714801aad0 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.124+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f71480193f0 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.125+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7154061c70 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.127+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f7148029060 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.259+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f715419cbc0 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.260+0000 7f7151ffb700 1 -- 172.21.15.64:0/1692063256 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f7148033090 con 0x7f71541052f0 2023-07-14T18:16:20.686 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 msgr2=0x7f7154199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 --2- 172.21.15.64:0/1692063256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7148011dc0 tx=0x7f7148011ea0 comp rx=0 tx=0).stop 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 shutdown_connections 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 --2- 172.21.15.64:0/1692063256 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71541052f0 0x7f7154199020 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 >> 172.21.15.64:0/1692063256 conn(0x7f7154100960 msgr2=0x7f7154102030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 shutdown_connections 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:20.261+0000 7f715af0c700 1 -- 172.21.15.64:0/1692063256 wait complete. 2023-07-14T18:16:20.687 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (2/15)... 2023-07-14T18:16:21.042 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:20 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:20.596+0000 7f32dda81200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:16:21.042 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:20 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:20.755+0000 7f32dda81200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:16:21.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:20 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:20.996+0000 7f32dda81200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:16:21.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:21 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:21.134+0000 7f32dda81200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:16:21.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:21 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:21.406+0000 7f32dda81200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:16:22.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:22.448+0000 7f32dda81200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:16:22.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: Activating manager daemon x 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: mgrmap e2: x(active, starting, since 0.00309821s) 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:16:22.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: Manager daemon x is now available 2023-07-14T18:16:22.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:16:22.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:16:22.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' 2023-07-14T18:16:22.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:22 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' 2023-07-14T18:16:23.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:23 smithi064 ceph-mon[96720]: from='mgr.14100 172.21.15.64:0/3175983912' entity='mgr.x' 2023-07-14T18:16:23.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:23 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3499039984' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:16:23.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:23 smithi064 ceph-mon[96720]: mgrmap e3: x(active, since 1.00565s) 2023-07-14T18:16:23.716 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:16:23.717 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:16:23.718 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.719 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:16:23.720 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:23.720 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:16:23.720 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:16:23.725 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:16:23.725 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:16:23.726 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:16:23.727 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:16:23.728 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:16:23.729 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:16:04.094856+0000", 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.117+0000 7f2b7b904700 1 Processor -- start 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.117+0000 7f2b7b904700 1 -- start start 2023-07-14T18:16:23.730 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.117+0000 7f2b7b904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.117+0000 7f2b7b904700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b740796c0 con 0x7f2b7407ac80 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b796a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b796a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45548/0 (socket says 172.21.15.64:45548) 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b796a0700 1 -- 172.21.15.64:0/3727206031 learned_addr learned my addr 172.21.15.64:0/3727206031 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b796a0700 1 -- 172.21.15.64:0/3727206031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b74079800 con 0x7f2b7407ac80 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b796a0700 1 --2- 172.21.15.64:0/3727206031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b640098a0 tx=0x7f2b64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53eabf42e1bbef0b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:23.731 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b6bfff700 1 -- 172.21.15.64:0/3727206031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b64004160 con 0x7f2b7407ac80 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b6bfff700 1 -- 172.21.15.64:0/3727206031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2b640042c0 con 0x7f2b7407ac80 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.118+0000 7f2b6bfff700 1 -- 172.21.15.64:0/3727206031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b64011940 con 0x7f2b7407ac80 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 -- 172.21.15.64:0/3727206031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 msgr2=0x7f2b74079180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 --2- 172.21.15.64:0/3727206031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2b640098a0 tx=0x7f2b64009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 -- 172.21.15.64:0/3727206031 shutdown_connections 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 --2- 172.21.15.64:0/3727206031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b74079180 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 -- 172.21.15.64:0/3727206031 >> 172.21.15.64:0/3727206031 conn(0x7f2b74100960 msgr2=0x7f2b74102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:23.732 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 -- 172.21.15.64:0/3727206031 shutdown_connections 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.119+0000 7f2b7b904700 1 -- 172.21.15.64:0/3727206031 wait complete. 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b7b904700 1 Processor -- start 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b7b904700 1 -- start start 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b7b904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b7b904700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7419b7c0 con 0x7f2b7407ac80 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b796a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b796a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45550/0 (socket says 172.21.15.64:45550) 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b796a0700 1 -- 172.21.15.64:0/3499039984 learned_addr learned my addr 172.21.15.64:0/3499039984 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b796a0700 1 -- 172.21.15.64:0/3499039984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b64009550 con 0x7f2b7407ac80 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b796a0700 1 --2- 172.21.15.64:0/3499039984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64004250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b640036a0 con 0x7f2b7407ac80 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.120+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2b64003c70 con 0x7f2b7407ac80 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.121+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b7419b9c0 con 0x7f2b7407ac80 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.121+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b64024d70 con 0x7f2b7407ac80 2023-07-14T18:16:23.734 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.121+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7419bde0 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.122+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f2b6401a430 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.122+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2b6404c2f0 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.122+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b74061c70 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.124+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f2b640245d0 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.256+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2b7419e910 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.256+0000 7f2b6a7fc700 1 -- 172.21.15.64:0/3499039984 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f2b6402cca0 con 0x7f2b7407ac80 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.257+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 msgr2=0x7f2b7419b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:23.735 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.257+0000 7f2b7b904700 1 --2- 172.21.15.64:0/3499039984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64004250 comp rx=0 tx=0).stop 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.257+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 shutdown_connections 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.257+0000 7f2b7b904700 1 --2- 172.21.15.64:0/3499039984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2b7407ac80 0x7f2b7419b280 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.257+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 >> 172.21.15.64:0/3499039984 conn(0x7f2b74100960 msgr2=0x7f2b74102030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.258+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 shutdown_connections 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:23.258+0000 7f2b7b904700 1 -- 172.21.15.64:0/3499039984 wait complete. 2023-07-14T18:16:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:mgr not available, waiting (3/15)... 2023-07-14T18:16:25.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:25 smithi064 ceph-mon[96720]: mgrmap e4: x(active, since 2s) 2023-07-14T18:16:27.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/675446190' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:16:27.171 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:27.171 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:27.171 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:16:27.171 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:16:27.172 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:16:27.172 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:16:27.172 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:16:27.172 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.172 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:27.173 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "quorum_age": 18, 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:27.174 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:16:27.175 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:16:27.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:16:27.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:16:27.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:16:27.181 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:16:27.182 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:16:27.183 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:16:27.184 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:16:27.185 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:16:04.094856+0000", 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f4405bed700 1 Processor -- start 2023-07-14T18:16:27.186 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f4405bed700 1 -- start start 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f4405bed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f4405bed700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4400105c70 con 0x7f4400073b60 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45562/0 (socket says 172.21.15.64:45562) 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f43ff7fe700 1 -- 172.21.15.64:0/3701215315 learned_addr learned my addr 172.21.15.64:0/3701215315 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.198+0000 7f43ff7fe700 1 -- 172.21.15.64:0/3701215315 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440010a7c0 con 0x7f4400073b60 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f43ff7fe700 1 --2- 172.21.15.64:0/3701215315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43f00098a0 tx=0x7f43f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4b8edf81f70e2d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:27.187 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f43fe7fc700 1 -- 172.21.15.64:0/3701215315 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43f0004160 con 0x7f4400073b60 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f43fe7fc700 1 -- 172.21.15.64:0/3701215315 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f43f00042c0 con 0x7f4400073b60 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f43fe7fc700 1 -- 172.21.15.64:0/3701215315 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43f0011940 con 0x7f4400073b60 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 -- 172.21.15.64:0/3701215315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 msgr2=0x7f440010a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 --2- 172.21.15.64:0/3701215315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43f00098a0 tx=0x7f43f0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 -- 172.21.15.64:0/3701215315 shutdown_connections 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 --2- 172.21.15.64:0/3701215315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400073b60 0x7f440010a280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 -- 172.21.15.64:0/3701215315 >> 172.21.15.64:0/3701215315 conn(0x7f44000ff850 msgr2=0x7f4400101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 -- 172.21.15.64:0/3701215315 shutdown_connections 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.199+0000 7f4405bed700 1 -- 172.21.15.64:0/3701215315 wait complete. 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.200+0000 7f4405bed700 1 Processor -- start 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.200+0000 7f4405bed700 1 -- start start 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.200+0000 7f4405bed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.200+0000 7f4405bed700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019bb00 con 0x7f440019b1b0 2023-07-14T18:16:27.189 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43ff7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45564/0 (socket says 172.21.15.64:45564) 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43ff7fe700 1 -- 172.21.15.64:0/675446190 learned_addr learned my addr 172.21.15.64:0/675446190 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43ff7fe700 1 -- 172.21.15.64:0/675446190 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f0009550 con 0x7f440019b1b0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43ff7fe700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006af0 tx=0x7f43f0011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43f0003bf0 con 0x7f440019b1b0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f43f001b8b0 con 0x7f440019b1b0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440019bd00 con 0x7f440019b1b0 2023-07-14T18:16:27.190 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43f00198b0 con 0x7f440019b1b0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.201+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440019e960 con 0x7f440019b1b0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.202+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f43f0019b20 con 0x7f440019b1b0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.202+0000 7f43fcff9700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 0x7f43e803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.202+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4400073b60 con 0x7f440019b1b0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.202+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f43f004cbe0 con 0x7f440019b1b0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.202+0000 7f43feffd700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 0x7f43e803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:27.191 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.203+0000 7f43feffd700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 0x7f43e803ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.207+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f43f00187c0 con 0x7f440019b1b0 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.726+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f440019eaf0 con 0x7f440019b1b0 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.726+0000 7f43fcff9700 1 -- 172.21.15.64:0/675446190 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f43f00187c0 con 0x7f440019b1b0 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 msgr2=0x7f43e803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 0x7f43e803ab70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f43f4006fd0 tx=0x7f43f4006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 msgr2=0x7f440019b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f43f0006af0 tx=0x7f43f0011a30 comp rx=0 tx=0).stop 2023-07-14T18:16:27.192 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 shutdown_connections 2023-07-14T18:16:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f43e80386c0 0x7f43e803ab70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 --2- 172.21.15.64:0/675446190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f440019b1b0 0x7f440019b5c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.728+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 >> 172.21.15.64:0/675446190 conn(0x7f44000ff850 msgr2=0x7f4400100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.729+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 shutdown_connections 2023-07-14T18:16:27.193 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:26.729+0000 7f4405bed700 1 -- 172.21.15.64:0/675446190 wait complete. 2023-07-14T18:16:27.194 INFO:teuthology.orchestra.run.smithi064.stdout:mgr is available 2023-07-14T18:16:28.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:28 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4014475677' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout fsid = 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:16:28.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:16:28.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:16:28.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.643+0000 7f4b93c96700 1 Processor -- start 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b93c96700 1 -- start start 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b93c96700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b93c96700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c105ca0 con 0x7f4b8c073b60 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b91a32700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b91a32700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45568/0 (socket says 172.21.15.64:45568) 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b91a32700 1 -- 172.21.15.64:0/128445773 learned_addr learned my addr 172.21.15.64:0/128445773 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b91a32700 1 -- 172.21.15.64:0/128445773 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c10a7f0 con 0x7f4b8c073b60 2023-07-14T18:16:28.430 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.644+0000 7f4b91a32700 1 --2- 172.21.15.64:0/128445773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b880098a0 tx=0x7f4b88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6256a874c541fc4a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b90a30700 1 -- 172.21.15.64:0/128445773 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b88004160 con 0x7f4b8c073b60 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b90a30700 1 -- 172.21.15.64:0/128445773 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4b880042c0 con 0x7f4b8c073b60 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b90a30700 1 -- 172.21.15.64:0/128445773 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b88011940 con 0x7f4b8c073b60 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 -- 172.21.15.64:0/128445773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 msgr2=0x7f4b8c10a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 --2- 172.21.15.64:0/128445773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4b880098a0 tx=0x7f4b88009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 -- 172.21.15.64:0/128445773 shutdown_connections 2023-07-14T18:16:28.431 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 --2- 172.21.15.64:0/128445773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c073b60 0x7f4b8c10a2b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 -- 172.21.15.64:0/128445773 >> 172.21.15.64:0/128445773 conn(0x7f4b8c0ff880 msgr2=0x7f4b8c101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 -- 172.21.15.64:0/128445773 shutdown_connections 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.645+0000 7f4b93c96700 1 -- 172.21.15.64:0/128445773 wait complete. 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b93c96700 1 Processor -- start 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b93c96700 1 -- start start 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b93c96700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b93c96700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c19bb80 con 0x7f4b8c19b230 2023-07-14T18:16:28.432 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b91a32700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b91a32700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45570/0 (socket says 172.21.15.64:45570) 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.646+0000 7f4b91a32700 1 -- 172.21.15.64:0/4014475677 learned_addr learned my addr 172.21.15.64:0/4014475677 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b91a32700 1 -- 172.21.15.64:0/4014475677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88009550 con 0x7f4b8c19b230 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b91a32700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4b88000c00 tx=0x7f4b88011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b880036a0 con 0x7f4b8c19b230 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4b88003c70 con 0x7f4b8c19b230 2023-07-14T18:16:28.433 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c19bd80 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4b880193f0 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.647+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c19e9e0 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.648+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f4b88003800 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.648+0000 7f4b7effd700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 0x7f4b7803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.648+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4b8804c470 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.648+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c073b60 con 0x7f4b8c19b230 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.648+0000 7f4b91231700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 0x7f4b7803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:28.434 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.649+0000 7f4b91231700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 0x7f4b7803abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b80006fd0 tx=0x7f4b80006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:27.653+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4b8801f020 con 0x7f4b8c19b230 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.006+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4b8c19bf10 con 0x7f4b8c19b230 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.007+0000 7f4b7effd700 1 -- 172.21.15.64:0/4014475677 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f4b8804a020 con 0x7f4b8c19b230 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 msgr2=0x7f4b7803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 0x7f4b7803abc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4b80006fd0 tx=0x7f4b80006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 msgr2=0x7f4b8c19b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4b88000c00 tx=0x7f4b88011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:28.435 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 shutdown_connections 2023-07-14T18:16:28.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f4b78038710 0x7f4b7803abc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:28.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 --2- 172.21.15.64:0/4014475677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b8c19b230 0x7f4b8c19b640 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:28.436 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 >> 172.21.15.64:0/4014475677 conn(0x7f4b8c0ff880 msgr2=0x7f4b8c100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:28.437 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 shutdown_connections 2023-07-14T18:16:28.437 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.009+0000 7f4b93c96700 1 -- 172.21.15.64:0/4014475677 wait complete. 2023-07-14T18:16:28.437 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling cephadm module... 2023-07-14T18:16:29.830 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1530608752' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T18:16:29.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: ignoring --setuser ceph since I am not root 2023-07-14T18:16:29.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: ignoring --setgroup ceph since I am not root 2023-07-14T18:16:29.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:29.690+0000 7f600765d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:16:29.831 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:29.786+0000 7f600765d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.896+0000 7f2f75593700 1 Processor -- start 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.896+0000 7f2f75593700 1 -- start start 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f75593700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f75593700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f70104b20 con 0x7f2f701041d0 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:29.973 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45574/0 (socket says 172.21.15.64:45574) 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6effd700 1 -- 172.21.15.64:0/404263930 learned_addr learned my addr 172.21.15.64:0/404263930 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6effd700 1 -- 172.21.15.64:0/404263930 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f70104c60 con 0x7f2f701041d0 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6effd700 1 --2- 172.21.15.64:0/404263930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2f600098a0 tx=0x7f2f60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cd9ecc13b3a473 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.897+0000 7f2f6dffb700 1 -- 172.21.15.64:0/404263930 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f60004160 con 0x7f2f701041d0 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f6dffb700 1 -- 172.21.15.64:0/404263930 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2f600042c0 con 0x7f2f701041d0 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f6dffb700 1 -- 172.21.15.64:0/404263930 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f60011940 con 0x7f2f701041d0 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 -- 172.21.15.64:0/404263930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 msgr2=0x7f2f701045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:29.974 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 --2- 172.21.15.64:0/404263930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2f600098a0 tx=0x7f2f60009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 -- 172.21.15.64:0/404263930 shutdown_connections 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 --2- 172.21.15.64:0/404263930 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f701045e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 -- 172.21.15.64:0/404263930 >> 172.21.15.64:0/404263930 conn(0x7f2f700ff840 msgr2=0x7f2f70101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 -- 172.21.15.64:0/404263930 shutdown_connections 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.898+0000 7f2f75593700 1 -- 172.21.15.64:0/404263930 wait complete. 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.899+0000 7f2f75593700 1 Processor -- start 2023-07-14T18:16:29.975 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.899+0000 7f2f75593700 1 -- start start 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.899+0000 7f2f75593700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.899+0000 7f2f75593700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f701983a0 con 0x7f2f701041d0 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f6effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45576/0 (socket says 172.21.15.64:45576) 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f6effd700 1 -- 172.21.15.64:0/1530608752 learned_addr learned my addr 172.21.15.64:0/1530608752 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f6effd700 1 -- 172.21.15.64:0/1530608752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f60009550 con 0x7f2f701041d0 2023-07-14T18:16:29.976 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f6effd700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f60011dc0 tx=0x7f2f60011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f600037d0 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f701985a0 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2f6001a560 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f6000b570 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.900+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f701989c0 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.901+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f2f60003930 con 0x7f2f701041d0 2023-07-14T18:16:29.977 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.901+0000 7f2f67fff700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 0x7f2f5c03ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.901+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2f6004dd40 con 0x7f2f701041d0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.901+0000 7f2f6e7fc700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 0x7f2f5c03ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.902+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f70061c70 con 0x7f2f701041d0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.902+0000 7f2f6e7fc700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 0x7f2f5c03ab60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f58006fd0 tx=0x7f2f58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:28.907+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2f6001a6d0 con 0x7f2f701041d0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.372+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2f700733e0 con 0x7f2f701041d0 2023-07-14T18:16:29.978 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.505+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2f6001aa40 con 0x7f2f701041d0 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.506+0000 7f2f67fff700 1 -- 172.21.15.64:0/1530608752 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2f6004d760 con 0x7f2f701041d0 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 msgr2=0x7f2f5c03ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 0x7f2f5c03ab60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2f58006fd0 tx=0x7f2f58006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 msgr2=0x7f2f70197e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2f60011dc0 tx=0x7f2f60011ea0 comp rx=0 tx=0).stop 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 shutdown_connections 2023-07-14T18:16:29.979 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7f2f5c0386b0 0x7f2f5c03ab60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:29.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 --2- 172.21.15.64:0/1530608752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f701041d0 0x7f2f70197e60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:29.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 >> 172.21.15.64:0/1530608752 conn(0x7f2f700ff840 msgr2=0x7f2f70100600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:29.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 shutdown_connections 2023-07-14T18:16:29.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:29.509+0000 7f2f75593700 1 -- 172.21.15.64:0/1530608752 wait complete. 2023-07-14T18:16:30.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:29 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:29.894+0000 7f600765d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:16:30.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:30.011+0000 7f600765d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:16:30.549 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:30.223+0000 7f600765d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:16:30.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1530608752' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T18:16:30.848 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-mon[96720]: mgrmap e5: x(active, since 7s) 2023-07-14T18:16:30.848 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:30.620+0000 7f600765d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:16:30.848 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:30.728+0000 7f600765d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:16:31.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:30 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:30.881+0000 7f600765d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:16:31.336 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:31.000+0000 7f600765d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:16:31.420 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:31.420 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.449+0000 7fced81ae700 1 Processor -- start 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.449+0000 7fced81ae700 1 -- start start 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.449+0000 7fced81ae700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:31.421 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.449+0000 7fced81ae700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0073be0 con 0x7fced0105df0 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced5f4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced5f4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45590/0 (socket says 172.21.15.64:45590) 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced5f4a700 1 -- 172.21.15.64:0/2198559609 learned_addr learned my addr 172.21.15.64:0/2198559609 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced5f4a700 1 -- 172.21.15.64:0/2198559609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0073d20 con 0x7fced0105df0 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced5f4a700 1 --2- 172.21.15.64:0/2198559609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0098a0 tx=0x7fcecc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671674ddca4cd283 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced4f48700 1 -- 172.21.15.64:0/2198559609 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc004160 con 0x7fced0105df0 2023-07-14T18:16:31.422 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced4f48700 1 -- 172.21.15.64:0/2198559609 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fcecc0042c0 con 0x7fced0105df0 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.450+0000 7fced4f48700 1 -- 172.21.15.64:0/2198559609 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc011940 con 0x7fced0105df0 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 -- 172.21.15.64:0/2198559609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 msgr2=0x7fced01081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 --2- 172.21.15.64:0/2198559609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcecc0098a0 tx=0x7fcecc009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 -- 172.21.15.64:0/2198559609 shutdown_connections 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 --2- 172.21.15.64:0/2198559609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced01081d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 -- 172.21.15.64:0/2198559609 >> 172.21.15.64:0/2198559609 conn(0x7fced00ff850 msgr2=0x7fced0101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 -- 172.21.15.64:0/2198559609 shutdown_connections 2023-07-14T18:16:31.423 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.451+0000 7fced81ae700 1 -- 172.21.15.64:0/2198559609 wait complete. 2023-07-14T18:16:31.424 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced81ae700 1 Processor -- start 2023-07-14T18:16:31.424 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced81ae700 1 -- start start 2023-07-14T18:16:31.424 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced81ae700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced81ae700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced0105c90 con 0x7fced0105df0 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced5f4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced5f4a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45592/0 (socket says 172.21.15.64:45592) 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced5f4a700 1 -- 172.21.15.64:0/932424568 learned_addr learned my addr 172.21.15.64:0/932424568 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced5f4a700 1 -- 172.21.15.64:0/932424568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcecc009550 con 0x7fced0105df0 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.452+0000 7fced5f4a700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006af0 tx=0x7fcecc011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.453+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc003bf0 con 0x7fced0105df0 2023-07-14T18:16:31.425 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.453+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fcecc01b8b0 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.453+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcecc0198b0 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.453+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced0103da0 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.453+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced01041c0 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fcecc019b20 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fcec6ffd700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 0x7fcebc03abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced0061c70 con 0x7fced0105df0 2023-07-14T18:16:31.426 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fced5749700 1 -- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 msgr2=0x7fcebc03abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fcecc04cbc0 con 0x7fced0105df0 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.454+0000 7fced5749700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 0x7fcebc03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.459+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcecc018530 con 0x7fced0105df0 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.655+0000 7fced5749700 1 -- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 msgr2=0x7fcebc03abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.655+0000 7fced5749700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 0x7fcebc03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.923+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fced01044c0 con 0x7fced0105df0 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.923+0000 7fcec6ffd700 1 -- 172.21.15.64:0/932424568 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fcecc018350 con 0x7fced0105df0 2023-07-14T18:16:31.427 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.925+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 msgr2=0x7fcebc03abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.925+0000 7fced81ae700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 0x7fcebc03abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 msgr2=0x7fced0105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fcecc006af0 tx=0x7fcecc011a30 comp rx=0 tx=0).stop 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 shutdown_connections 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7fcebc038710 0x7fcebc03abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 --2- 172.21.15.64:0/932424568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fced0105df0 0x7fced0105750 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:31.428 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 >> 172.21.15.64:0/932424568 conn(0x7fced00ff850 msgr2=0x7fced0100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:31.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 shutdown_connections 2023-07-14T18:16:31.429 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:30.926+0000 7fced81ae700 1 -- 172.21.15.64:0/932424568 wait complete. 2023-07-14T18:16:31.429 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2023-07-14T18:16:31.429 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 5... 2023-07-14T18:16:31.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:31 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/932424568' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:16:31.634 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:31.517+0000 7f600765d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:16:32.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:32 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:32.205+0000 7f600765d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:16:32.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:32 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:32.396+0000 7f600765d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:16:33.724 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:33 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:33.353+0000 7f600765d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:16:33.725 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:33 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:33.463+0000 7f600765d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:16:34.069 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:33 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:33.680+0000 7f600765d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:16:34.322 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:34 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:34.025+0000 7f600765d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:16:34.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:34 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:34.278+0000 7f600765d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:16:34.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:34 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:34.458+0000 7f600765d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:16:35.085 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:34 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:34.783+0000 7f600765d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:16:36.051 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:35 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:35.604+0000 7f600765d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:16:36.051 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:35 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:35.761+0000 7f600765d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:16:36.051 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:36 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:36.006+0000 7f600765d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:16:36.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:36 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:36.145+0000 7f600765d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:16:36.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:36 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:36.417+0000 7f600765d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:16:37.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:37.474+0000 7f600765d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:16:37.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: Active manager daemon x restarted 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: Activating manager daemon x 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: mgrmap e6: x(active, starting, since 0.00358742s) 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:16:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:16:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:16:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: Manager daemon x is now available 2023-07-14T18:16:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:37 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:38.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: Found migration_current of "None". Setting to last migration. 2023-07-14T18:16:38.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:38 smithi064 ceph-mon[96720]: mgrmap e7: x(active, since 1.00657s) 2023-07-14T18:16:38.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:38.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T18:16:38.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:16:38.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:38.980 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.878+0000 7ff35f1c6700 1 Processor -- start 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.878+0000 7ff35f1c6700 1 -- start start 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.878+0000 7ff35f1c6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.878+0000 7ff35f1c6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff358073ba0 con 0x7ff358105e30 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff35cf62700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff35cf62700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45600/0 (socket says 172.21.15.64:45600) 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff35cf62700 1 -- 172.21.15.64:0/292528003 learned_addr learned my addr 172.21.15.64:0/292528003 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:38.981 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff35cf62700 1 -- 172.21.15.64:0/292528003 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358073ce0 con 0x7ff358105e30 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff35cf62700 1 --2- 172.21.15.64:0/292528003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0098a0 tx=0x7ff34c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92dc6acf3e8c025 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff3577fe700 1 -- 172.21.15.64:0/292528003 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff34c004160 con 0x7ff358105e30 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.879+0000 7ff3577fe700 1 -- 172.21.15.64:0/292528003 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff34c0042c0 con 0x7ff358105e30 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff3577fe700 1 -- 172.21.15.64:0/292528003 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff34c011940 con 0x7ff358105e30 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 -- 172.21.15.64:0/292528003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 msgr2=0x7ff358108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/292528003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0098a0 tx=0x7ff34c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.982 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 -- 172.21.15.64:0/292528003 shutdown_connections 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/292528003 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff358108210 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 -- 172.21.15.64:0/292528003 >> 172.21.15.64:0/292528003 conn(0x7ff3580ff850 msgr2=0x7ff358101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 -- 172.21.15.64:0/292528003 shutdown_connections 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.880+0000 7ff35f1c6700 1 -- 172.21.15.64:0/292528003 wait complete. 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35f1c6700 1 Processor -- start 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35f1c6700 1 -- start start 2023-07-14T18:16:38.983 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35f1c6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35f1c6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35819b6c0 con 0x7ff358105e30 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35cf62700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35cf62700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45602/0 (socket says 172.21.15.64:45602) 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.881+0000 7ff35cf62700 1 -- 172.21.15.64:0/2962260721 learned_addr learned my addr 172.21.15.64:0/2962260721 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff35cf62700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34c009550 con 0x7ff358105e30 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff35cf62700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:38.984 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff34c0036a0 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7ff34c01b440 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35819b8c0 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff34c0193f0 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.882+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35819bd40 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7ff34c019670 con 0x7ff358105e30 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff355ffb700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:38.985 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff358061c70 con 0x7ff348038710 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff34c04c620 con 0x7ff358105e30 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff357fff700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:31.883+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:32.084+0000 7ff357fff700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:32.084+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:32.485+0000 7ff357fff700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:38.986 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:32.485+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:33.286+0000 7ff357fff700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:33.286+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:34.887+0000 7ff357fff700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/3374567519 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:34.887+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:37.489+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7ff34c018c70 con 0x7ff358105e30 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:37.489+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 msgr2=0x7ff34803abc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:16:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:37.489+0000 7ff355ffb700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.491+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7ff34c04de30 con 0x7ff358105e30 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.491+0000 7ff355ffb700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 0x7ff34803e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.491+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff358061c70 con 0x7ff34803be30 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.491+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 0x7ff34803e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.492+0000 7ff357fff700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 0x7ff34803e210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff344003a10 tx=0x7ff3440092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.493+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7ff358061c70 con 0x7ff34803be30 2023-07-14T18:16:38.988 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.503+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff35819c4c0 con 0x7ff34803be30 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff355ffb700 1 -- 172.21.15.64:0/2962260721 <== mgr.14120 v2:172.21.15.64:6800/792928752 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff35819c4c0 con 0x7ff34803be30 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 msgr2=0x7ff34803e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 0x7ff34803e210 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff344003a10 tx=0x7ff3440092b0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 msgr2=0x7ff35819b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff34c000c00 tx=0x7ff34c011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 shutdown_connections 2023-07-14T18:16:38.989 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7ff34803be30 0x7ff34803e210 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff358105e30 0x7ff35819b180 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 --2- 172.21.15.64:0/2962260721 >> [v2:172.21.15.64:6800/3374567519,v1:172.21.15.64:6801/3374567519] conn(0x7ff348038710 0x7ff34803abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 >> 172.21.15.64:0/2962260721 conn(0x7ff3580ff850 msgr2=0x7ff358100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 shutdown_connections 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:38.504+0000 7ff35f1c6700 1 -- 172.21.15.64:0/2962260721 wait complete. 2023-07-14T18:16:38.990 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 5 is available 2023-07-14T18:16:38.991 INFO:teuthology.orchestra.run.smithi064.stdout:Setting orchestrator backend to cephadm... 2023-07-14T18:16:39.904 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:39 smithi064 ceph-mon[96720]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:16:39.904 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:39 smithi064 ceph-mon[96720]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:16:40.321 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.418+0000 7f488af52700 1 Processor -- start 2023-07-14T18:16:40.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.418+0000 7f488af52700 1 -- start start 2023-07-14T18:16:40.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.418+0000 7f488af52700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:40.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.418+0000 7f488af52700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840796a0 con 0x7f488407ac60 2023-07-14T18:16:40.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.419+0000 7f4888cee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:40.326 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.419+0000 7f4888cee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45646/0 (socket says 172.21.15.64:45646) 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.419+0000 7f4888cee700 1 -- 172.21.15.64:0/4030842739 learned_addr learned my addr 172.21.15.64:0/4030842739 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.419+0000 7f4888cee700 1 -- 172.21.15.64:0/4030842739 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840797e0 con 0x7f488407ac60 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.419+0000 7f4888cee700 1 --2- 172.21.15.64:0/4030842739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48780098a0 tx=0x7f4878009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412bde05c94c4789 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f48837fe700 1 -- 172.21.15.64:0/4030842739 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4878004160 con 0x7f488407ac60 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f48837fe700 1 -- 172.21.15.64:0/4030842739 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f48780042c0 con 0x7f488407ac60 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f48837fe700 1 -- 172.21.15.64:0/4030842739 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4878011940 con 0x7f488407ac60 2023-07-14T18:16:40.327 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 -- 172.21.15.64:0/4030842739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 msgr2=0x7f4884079160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 --2- 172.21.15.64:0/4030842739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f48780098a0 tx=0x7f4878009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 -- 172.21.15.64:0/4030842739 shutdown_connections 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 --2- 172.21.15.64:0/4030842739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f4884079160 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 -- 172.21.15.64:0/4030842739 >> 172.21.15.64:0/4030842739 conn(0x7f4884100950 msgr2=0x7f4884102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 -- 172.21.15.64:0/4030842739 shutdown_connections 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.420+0000 7f488af52700 1 -- 172.21.15.64:0/4030842739 wait complete. 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.421+0000 7f488af52700 1 Processor -- start 2023-07-14T18:16:40.328 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.421+0000 7f488af52700 1 -- start start 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f488af52700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f488af52700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488419b730 con 0x7f488407ac60 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4888cee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4888cee700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45648/0 (socket says 172.21.15.64:45648) 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4888cee700 1 -- 172.21.15.64:0/1019545032 learned_addr learned my addr 172.21.15.64:0/1019545032 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4888cee700 1 -- 172.21.15.64:0/1019545032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4878009550 con 0x7f488407ac60 2023-07-14T18:16:40.329 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4888cee700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4878000c00 tx=0x7f4878011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48780036a0 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488419b930 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4878003c70 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.422+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f48780193f0 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.423+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488419bd50 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.424+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4884061c70 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.424+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7f487801b440 con 0x7f488407ac60 2023-07-14T18:16:40.330 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.424+0000 7f4881ffb700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 0x7f487403aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.424+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f487804c520 con 0x7f488407ac60 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.424+0000 7f4883fff700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 0x7f487403aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.425+0000 7f4883fff700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 0x7f487403aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4870006fd0 tx=0x7f4870006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.429+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4878003990 con 0x7f488407ac60 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.542+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f487804b840 con 0x7f488407ac60 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.824+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f4884102c20 con 0x7f48740385e0 2023-07-14T18:16:40.331 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.830+0000 7f4881ffb700 1 -- 172.21.15.64:0/1019545032 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4884102c20 con 0x7f48740385e0 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 msgr2=0x7f487403aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 0x7f487403aa90 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4870006fd0 tx=0x7f4870006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 msgr2=0x7f488419b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4878000c00 tx=0x7f4878011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 shutdown_connections 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f48740385e0 0x7f487403aa90 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 --2- 172.21.15.64:0/1019545032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f488407ac60 0x7f488419b1f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:40.332 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.832+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 >> 172.21.15.64:0/1019545032 conn(0x7f4884100950 msgr2=0x7f4884102000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:40.333 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.833+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 shutdown_connections 2023-07-14T18:16:40.333 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:39.833+0000 7f488af52700 1 -- 172.21.15.64:0/1019545032 wait complete. 2023-07-14T18:16:40.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:40 smithi064 ceph-mon[96720]: mgrmap e8: x(active, since 2s) 2023-07-14T18:16:40.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:40 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:40.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:40 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:40.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:40 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:41.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T18:16:41.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40b6726700 1 Processor -- start 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40b6726700 1 -- start start 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40b6726700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40b6726700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0073be0 con 0x7f40b0103c10 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45660/0 (socket says 172.21.15.64:45660) 2023-07-14T18:16:41.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.776+0000 7f40affff700 1 -- 172.21.15.64:0/2434802456 learned_addr learned my addr 172.21.15.64:0/2434802456 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.777+0000 7f40affff700 1 -- 172.21.15.64:0/2434802456 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0073d20 con 0x7f40b0103c10 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.777+0000 7f40affff700 1 --2- 172.21.15.64:0/2434802456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40a00098a0 tx=0x7f40a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1341f77252e647a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.777+0000 7f40aeffd700 1 -- 172.21.15.64:0/2434802456 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a0004160 con 0x7f40b0103c10 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.777+0000 7f40aeffd700 1 -- 172.21.15.64:0/2434802456 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f40a00042c0 con 0x7f40b0103c10 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.777+0000 7f40aeffd700 1 -- 172.21.15.64:0/2434802456 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a0011940 con 0x7f40b0103c10 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- 172.21.15.64:0/2434802456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 msgr2=0x7f40b0104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 --2- 172.21.15.64:0/2434802456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f40a00098a0 tx=0x7f40a0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:41.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- 172.21.15.64:0/2434802456 shutdown_connections 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 --2- 172.21.15.64:0/2434802456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b0104020 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- 172.21.15.64:0/2434802456 >> 172.21.15.64:0/2434802456 conn(0x7f40b00ff880 msgr2=0x7f40b0101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- 172.21.15.64:0/2434802456 shutdown_connections 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- 172.21.15.64:0/2434802456 wait complete. 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 Processor -- start 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.778+0000 7f40b6726700 1 -- start start 2023-07-14T18:16:41.574 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40b6726700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40b6726700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b019b7a0 con 0x7f40b0103c10 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45662/0 (socket says 172.21.15.64:45662) 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40affff700 1 -- 172.21.15.64:0/1435944378 learned_addr learned my addr 172.21.15.64:0/1435944378 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40affff700 1 -- 172.21.15.64:0/1435944378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a0009550 con 0x7f40b0103c10 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40affff700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f40a0000c00 tx=0x7f40a0011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a00036a0 con 0x7f40b0103c10 2023-07-14T18:16:41.575 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f40a0003c70 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.779+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40a00193f0 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.780+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b019b9a0 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.780+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b019bdc0 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.780+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f40a0019670 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.781+0000 7f40ad7fa700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 0x7f409803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.781+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b0194a90 con 0x7f40b0103c10 2023-07-14T18:16:41.576 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.781+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f40a004c580 con 0x7f40b0103c10 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.781+0000 7f40af7fe700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 0x7f409803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.781+0000 7f40af7fe700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 0x7f409803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f40a4006fd0 tx=0x7f40a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:40.786+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f40a0022be0 con 0x7f40b0103c10 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.141+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f40b0060a80 con 0x7f4098038710 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.142+0000 7f40ad7fa700 1 -- 172.21.15.64:0/1435944378 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f40b0060a80 con 0x7f4098038710 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 msgr2=0x7f409803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:41.577 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 0x7f409803abc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f40a4006fd0 tx=0x7f40a4006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 msgr2=0x7f40b019b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f40a0000c00 tx=0x7f40a0011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 shutdown_connections 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f4098038710 0x7f409803abc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 --2- 172.21.15.64:0/1435944378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f40b0103c10 0x7f40b019b260 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 >> 172.21.15.64:0/1435944378 conn(0x7f40b00ff880 msgr2=0x7f40b018f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 shutdown_connections 2023-07-14T18:16:41.578 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:41.144+0000 7f40b6726700 1 -- 172.21.15.64:0/1435944378 wait complete. 2023-07-14T18:16:41.579 INFO:teuthology.orchestra.run.smithi064.stdout:Generating ssh key... 2023-07-14T18:16:41.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:41 smithi064 ceph-mon[96720]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:41.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:41 smithi064 ceph-mon[96720]: [14/Jul/2023:18:16:40] ENGINE Bus STARTING 2023-07-14T18:16:41.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:41 smithi064 ceph-mon[96720]: [14/Jul/2023:18:16:40] ENGINE Serving on https://172.21.15.64:7150 2023-07-14T18:16:41.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:41 smithi064 ceph-mon[96720]: [14/Jul/2023:18:16:40] ENGINE Serving on http://172.21.15.64:8765 2023-07-14T18:16:41.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:41 smithi064 ceph-mon[96720]: [14/Jul/2023:18:16:40] ENGINE Bus STARTED 2023-07-14T18:16:42.925 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-mon[96720]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:42.925 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: Generating public/private rsa key pair. 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: Your identification has been saved in /tmp/tmpqs66t1k7/key. 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: Your public key has been saved in /tmp/tmpqs66t1k7/key.pub. 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: The key fingerprint is: 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: SHA256:TboHdTfif7Cu0aGM4qNew3lRm1t3e5G66/XAqjqKyu0 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: The key's randomart image is: 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: +---[RSA 3072]----+ 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | | 2023-07-14T18:16:42.926 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | | 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | o + o | 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | = + = ..| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | S o + +oo| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | . + + B.=+| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | B + =.*oo| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | . . . oo= +ooo| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: | ooE.+oo+..+=o .| 2023-07-14T18:16:42.927 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:42 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: +----[SHA256]-----+ 2023-07-14T18:16:43.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f0281b20700 1 Processor -- start 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f0281b20700 1 -- start start 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f0281b20700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f0281b20700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c104b30 con 0x7f027c1041e0 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45666/0 (socket says 172.21.15.64:45666) 2023-07-14T18:16:43.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f027b7fe700 1 -- 172.21.15.64:0/3019552976 learned_addr learned my addr 172.21.15.64:0/3019552976 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.005+0000 7f027b7fe700 1 -- 172.21.15.64:0/3019552976 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c104c70 con 0x7f027c1041e0 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f027b7fe700 1 --2- 172.21.15.64:0/3019552976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f026c00bb40 tx=0x7f026c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f46652ea02058278 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f027a7fc700 1 -- 172.21.15.64:0/3019552976 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f026c00dbe0 con 0x7f027c1041e0 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f027a7fc700 1 -- 172.21.15.64:0/3019552976 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f026c0044a0 con 0x7f027c1041e0 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f027a7fc700 1 -- 172.21.15.64:0/3019552976 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f026c003da0 con 0x7f027c1041e0 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 -- 172.21.15.64:0/3019552976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 msgr2=0x7f027c1045f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:43.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 --2- 172.21.15.64:0/3019552976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f026c00bb40 tx=0x7f026c00be50 comp rx=0 tx=0).stop 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 -- 172.21.15.64:0/3019552976 shutdown_connections 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 --2- 172.21.15.64:0/3019552976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c1045f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 -- 172.21.15.64:0/3019552976 >> 172.21.15.64:0/3019552976 conn(0x7f027c0ff850 msgr2=0x7f027c101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.006+0000 7f0281b20700 1 -- 172.21.15.64:0/3019552976 shutdown_connections 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.007+0000 7f0281b20700 1 -- 172.21.15.64:0/3019552976 wait complete. 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.007+0000 7f0281b20700 1 Processor -- start 2023-07-14T18:16:43.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.007+0000 7f0281b20700 1 -- start start 2023-07-14T18:16:43.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.007+0000 7f0281b20700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:43.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.007+0000 7f0281b20700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c19b750 con 0x7f027c1041e0 2023-07-14T18:16:43.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:43.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45668/0 (socket says 172.21.15.64:45668) 2023-07-14T18:16:43.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f027b7fe700 1 -- 172.21.15.64:0/3730129552 learned_addr learned my addr 172.21.15.64:0/3730129552 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f027b7fe700 1 -- 172.21.15.64:0/3730129552 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c00b7f0 con 0x7f027c1041e0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f027b7fe700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f026c003df0 tx=0x7f026c003ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f026c0104c0 con 0x7f027c1041e0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f026c010a90 con 0x7f027c1041e0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c19b950 con 0x7f027c1041e0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f026c021b80 con 0x7f027c1041e0 2023-07-14T18:16:43.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.008+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c19bd10 con 0x7f027c1041e0 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.009+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f026c0193f0 con 0x7f027c1041e0 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.009+0000 7f0278ff9700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 0x7f026403aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.009+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f026c04ca30 con 0x7f027c1041e0 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.009+0000 7f027affd700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 0x7f026403aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.009+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c190b10 con 0x7f027c1041e0 2023-07-14T18:16:43.225 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.010+0000 7f027affd700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 0x7f026403aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0270006fd0 tx=0x7f0270006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.014+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f026c00e360 con 0x7f027c1041e0 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.370+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f027c106150 con 0x7f02640386f0 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.686+0000 7f0278ff9700 1 -- 172.21.15.64:0/3730129552 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f027c106150 con 0x7f02640386f0 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 msgr2=0x7f026403aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 0x7f026403aba0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0270006fd0 tx=0x7f0270006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 msgr2=0x7f027c19b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f026c003df0 tx=0x7f026c003ed0 comp rx=0 tx=0).stop 2023-07-14T18:16:43.226 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 shutdown_connections 2023-07-14T18:16:43.227 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f02640386f0 0x7f026403aba0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:43.227 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 --2- 172.21.15.64:0/3730129552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f027c1041e0 0x7f027c19b210 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:43.227 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 >> 172.21.15.64:0/3730129552 conn(0x7f027c0ff850 msgr2=0x7f027c18dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:43.227 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 shutdown_connections 2023-07-14T18:16:43.227 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:42.689+0000 7f0281b20700 1 -- 172.21.15.64:0/3730129552 wait complete. 2023-07-14T18:16:43.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:43 smithi064 ceph-mon[96720]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:43.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:43 smithi064 ceph-mon[96720]: Generating ssh key... 2023-07-14T18:16:43.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:43 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:43.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:43 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:44.565 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.675+0000 7f2c9bca6700 1 Processor -- start 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.675+0000 7f2c9bca6700 1 -- start start 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.675+0000 7f2c9bca6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.675+0000 7f2c9bca6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94073be0 con 0x7f2c94105df0 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c99a42700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:44.566 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c99a42700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45672/0 (socket says 172.21.15.64:45672) 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c99a42700 1 -- 172.21.15.64:0/351484186 learned_addr learned my addr 172.21.15.64:0/351484186 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c99a42700 1 -- 172.21.15.64:0/351484186 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c94073d20 con 0x7f2c94105df0 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c99a42700 1 --2- 172.21.15.64:0/351484186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c840098a0 tx=0x7f2c84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f1b42bd7108a4d6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c98a40700 1 -- 172.21.15.64:0/351484186 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c84004160 con 0x7f2c94105df0 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c98a40700 1 -- 172.21.15.64:0/351484186 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2c840042c0 con 0x7f2c94105df0 2023-07-14T18:16:44.567 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.676+0000 7f2c98a40700 1 -- 172.21.15.64:0/351484186 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c84011940 con 0x7f2c94105df0 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 -- 172.21.15.64:0/351484186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 msgr2=0x7f2c941081d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/351484186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2c840098a0 tx=0x7f2c84009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 -- 172.21.15.64:0/351484186 shutdown_connections 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/351484186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c941081d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 -- 172.21.15.64:0/351484186 >> 172.21.15.64:0/351484186 conn(0x7f2c940ff850 msgr2=0x7f2c94101c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 -- 172.21.15.64:0/351484186 shutdown_connections 2023-07-14T18:16:44.568 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.677+0000 7f2c9bca6700 1 -- 172.21.15.64:0/351484186 wait complete. 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c9bca6700 1 Processor -- start 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c9bca6700 1 -- start start 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c9bca6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c9bca6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94105c90 con 0x7f2c94105df0 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c99a42700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c99a42700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45674/0 (socket says 172.21.15.64:45674) 2023-07-14T18:16:44.569 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c99a42700 1 -- 172.21.15.64:0/1810996495 learned_addr learned my addr 172.21.15.64:0/1810996495 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c99a42700 1 -- 172.21.15.64:0/1810996495 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c84009550 con 0x7f2c94105df0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.678+0000 7f2c99a42700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c84000c00 tx=0x7f2c84011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.679+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c840036a0 con 0x7f2c94105df0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.679+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2c8401b440 con 0x7f2c94105df0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.679+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c94103da0 con 0x7f2c94105df0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.679+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2c840193f0 con 0x7f2c94105df0 2023-07-14T18:16:44.570 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.679+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c941041c0 con 0x7f2c94105df0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2c84003800 con 0x7f2c94105df0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c94109380 con 0x7f2c94105df0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c8affd700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 0x7f2c8003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2c8404c600 con 0x7f2c94105df0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c99241700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 0x7f2c8003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.680+0000 7f2c99241700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 0x7f2c8003abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c90006fd0 tx=0x7f2c90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:44.571 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:43.685+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2c8401b940 con 0x7f2c94105df0 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.040+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c94101b20 con 0x7f2c80038710 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.041+0000 7f2c8affd700 1 -- 172.21.15.64:0/1810996495 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2c94101b20 con 0x7f2c80038710 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 msgr2=0x7f2c8003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 0x7f2c8003abc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2c90006fd0 tx=0x7f2c90006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 msgr2=0x7f2c94105750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c84000c00 tx=0x7f2c84011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:44.572 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 shutdown_connections 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f2c80038710 0x7f2c8003abc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 --2- 172.21.15.64:0/1810996495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c94105df0 0x7f2c94105750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.043+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 >> 172.21.15.64:0/1810996495 conn(0x7f2c940ff850 msgr2=0x7f2c94100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.044+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 shutdown_connections 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:44.044+0000 7f2c9bca6700 1 -- 172.21.15.64:0/1810996495 wait complete. 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:16:44.573 INFO:teuthology.orchestra.run.smithi064.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T18:16:44.574 INFO:teuthology.orchestra.run.smithi064.stdout:Adding host smithi064... 2023-07-14T18:16:44.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:44 smithi064 ceph-mon[96720]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:45.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:45 smithi064 ceph-mon[96720]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:47.085 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:46 smithi064 ceph-mon[96720]: Deploying cephadm binary to smithi064 2023-07-14T18:16:49.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:49 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:49.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:49 smithi064 ceph-mon[96720]: Added host smithi064 2023-07-14T18:16:49.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:49 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:16:50.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Added host 'smithi064' with addr '172.21.15.64' 2023-07-14T18:16:50.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.037+0000 7f0c4972e700 1 Processor -- start 2023-07-14T18:16:50.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.037+0000 7f0c4972e700 1 -- start start 2023-07-14T18:16:50.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.037+0000 7f0c4972e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:50.023 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.037+0000 7f0c4972e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c44104b20 con 0x7f0c441041d0 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.037+0000 7f0c42ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c42ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45678/0 (socket says 172.21.15.64:45678) 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c42ffd700 1 -- 172.21.15.64:0/248359460 learned_addr learned my addr 172.21.15.64:0/248359460 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c42ffd700 1 -- 172.21.15.64:0/248359460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c44104c60 con 0x7f0c441041d0 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c42ffd700 1 --2- 172.21.15.64:0/248359460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009e80 tx=0x7f0c3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b887e76fe2ba00d9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c41ffb700 1 -- 172.21.15.64:0/248359460 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c3400e070 con 0x7f0c441041d0 2023-07-14T18:16:50.024 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c41ffb700 1 -- 172.21.15.64:0/248359460 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f0c34003e90 con 0x7f0c441041d0 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.038+0000 7f0c41ffb700 1 -- 172.21.15.64:0/248359460 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c34004360 con 0x7f0c441041d0 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- 172.21.15.64:0/248359460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 msgr2=0x7f0c441045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 --2- 172.21.15.64:0/248359460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009e80 tx=0x7f0c3400b330 comp rx=0 tx=0).stop 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- 172.21.15.64:0/248359460 shutdown_connections 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 --2- 172.21.15.64:0/248359460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c441045e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- 172.21.15.64:0/248359460 >> 172.21.15.64:0/248359460 conn(0x7f0c440ff840 msgr2=0x7f0c44101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:50.025 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- 172.21.15.64:0/248359460 shutdown_connections 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- 172.21.15.64:0/248359460 wait complete. 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 Processor -- start 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.039+0000 7f0c4972e700 1 -- start start 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c4972e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c4972e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4418f830 con 0x7f0c441041d0 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c42ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c42ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45680/0 (socket says 172.21.15.64:45680) 2023-07-14T18:16:50.026 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c42ffd700 1 -- 172.21.15.64:0/2561464209 learned_addr learned my addr 172.21.15.64:0/2561464209 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c42ffd700 1 -- 172.21.15.64:0/2561464209 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34009b30 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c42ffd700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c34004000 tx=0x7f0c34003c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c3400e040 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4418d940 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f0c3400bc00 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c340126e0 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.040+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4418ddc0 con 0x7f0c441041d0 2023-07-14T18:16:50.027 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.041+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0c340129a0 con 0x7f0c441041d0 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.042+0000 7f0c3bfff700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 0x7f0c3003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.042+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0c3404c8c0 con 0x7f0c441041d0 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.042+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c44061c70 con 0x7f0c441041d0 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.042+0000 7f0c427fc700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 0x7f0c3003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.042+0000 7f0c427fc700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 0x7f0c3003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c006fd0 tx=0x7f0c2c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:50.028 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.046+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c34020020 con 0x7f0c441041d0 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:45.411+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi064", "addr": "172.21.15.64", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c4418e7c0 con 0x7f0c30038710 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.366+0000 7f0c3bfff700 1 -- 172.21.15.64:0/2561464209 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f0c4418e7c0 con 0x7f0c30038710 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.368+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 msgr2=0x7f0c3003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.368+0000 7f0c4972e700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 0x7f0c3003abc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c006fd0 tx=0x7f0c2c006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 msgr2=0x7f0c4418f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0c34004000 tx=0x7f0c34003c50 comp rx=0 tx=0).stop 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 shutdown_connections 2023-07-14T18:16:50.029 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0c30038710 0x7f0c3003abc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:50.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 --2- 172.21.15.64:0/2561464209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c441041d0 0x7f0c4418f2f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:50.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 >> 172.21.15.64:0/2561464209 conn(0x7f0c440ff840 msgr2=0x7f0c440736e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:50.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 shutdown_connections 2023-07-14T18:16:50.030 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:48.369+0000 7f0c4972e700 1 -- 172.21.15.64:0/2561464209 wait complete. 2023-07-14T18:16:50.030 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mon service... 2023-07-14T18:16:51.669 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.479+0000 7f5e8c9bc700 1 Processor -- start 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8c9bc700 1 -- start start 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8c9bc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8c9bc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88073be0 con 0x7f5e88105de0 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45714/0 (socket says 172.21.15.64:45714) 2023-07-14T18:16:51.670 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.480+0000 7f5e8659c700 1 -- 172.21.15.64:0/1781078693 learned_addr learned my addr 172.21.15.64:0/1781078693 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8659c700 1 -- 172.21.15.64:0/1781078693 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88073d20 con 0x7f5e88105de0 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8659c700 1 --2- 172.21.15.64:0/1781078693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098a0 tx=0x7f5e78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ae380d630b26555 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8559a700 1 -- 172.21.15.64:0/1781078693 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e78004160 con 0x7f5e88105de0 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8559a700 1 -- 172.21.15.64:0/1781078693 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5e780042c0 con 0x7f5e88105de0 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8559a700 1 -- 172.21.15.64:0/1781078693 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e78011940 con 0x7f5e88105de0 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/1781078693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 msgr2=0x7f5e881081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:51.671 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.481+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/1781078693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5e780098a0 tx=0x7f5e78009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/1781078693 shutdown_connections 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/1781078693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e881081c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/1781078693 >> 172.21.15.64:0/1781078693 conn(0x7f5e880ff840 msgr2=0x7f5e88101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/1781078693 shutdown_connections 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/1781078693 wait complete. 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.482+0000 7f5e8c9bc700 1 Processor -- start 2023-07-14T18:16:51.672 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8c9bc700 1 -- start start 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8c9bc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8c9bc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88199510 con 0x7f5e88105de0 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45716/0 (socket says 172.21.15.64:45716) 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8659c700 1 -- 172.21.15.64:0/2984436241 learned_addr learned my addr 172.21.15.64:0/2984436241 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8659c700 1 -- 172.21.15.64:0/2984436241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e78009550 con 0x7f5e88105de0 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8659c700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:51.673 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e780036a0 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5e78003c70 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e780193f0 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.483+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e88199710 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.484+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e88199b30 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.484+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f5e78019620 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.485+0000 7f5e777fe700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 0x7f5e7003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.485+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5e7804c560 con 0x7f5e88105de0 2023-07-14T18:16:51.674 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.485+0000 7f5e85d9b700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 0x7f5e7003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.485+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88061c70 con 0x7f5e88105de0 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.485+0000 7f5e85d9b700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 0x7f5e7003ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c006fd0 tx=0x7f5e7c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.490+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5e7801f020 con 0x7f5e88105de0 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.876+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f5e88101b10 con 0x7f5e70038760 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.881+0000 7f5e777fe700 1 -- 172.21.15.64:0/2984436241 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5e88101b10 con 0x7f5e70038760 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 msgr2=0x7f5e7003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:51.675 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 0x7f5e7003ac10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c006fd0 tx=0x7f5e7c006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 msgr2=0x7f5e88198fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5e78000c00 tx=0x7f5e78011e70 comp rx=0 tx=0).stop 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 shutdown_connections 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f5e70038760 0x7f5e7003ac10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 --2- 172.21.15.64:0/2984436241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e88105de0 0x7f5e88198fd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.883+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 >> 172.21.15.64:0/2984436241 conn(0x7f5e880ff840 msgr2=0x7f5e88100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.884+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 shutdown_connections 2023-07-14T18:16:51.676 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:50.884+0000 7f5e8c9bc700 1 -- 172.21.15.64:0/2984436241 wait complete. 2023-07-14T18:16:51.677 INFO:teuthology.orchestra.run.smithi064.stdout:Deploying unmanaged mgr service... 2023-07-14T18:16:52.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:51 smithi064 ceph-mon[96720]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:52.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:51 smithi064 ceph-mon[96720]: Saving service mon spec with placement count:5 2023-07-14T18:16:52.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:51 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:52.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:51 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:52.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:51 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T18:16:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f01003f2700 1 Processor -- start 2023-07-14T18:16:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f01003f2700 1 -- start start 2023-07-14T18:16:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f01003f2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:53.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f01003f2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8073be0 con 0x7f00f8105e00 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f00fe18e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f00fe18e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45720/0 (socket says 172.21.15.64:45720) 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.119+0000 7f00fe18e700 1 -- 172.21.15.64:0/2249804666 learned_addr learned my addr 172.21.15.64:0/2249804666 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.120+0000 7f00fe18e700 1 -- 172.21.15.64:0/2249804666 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f8073d20 con 0x7f00f8105e00 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.120+0000 7f00fe18e700 1 --2- 172.21.15.64:0/2249804666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00f40098a0 tx=0x7f00f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7fa2869712df98 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.120+0000 7f00fd18c700 1 -- 172.21.15.64:0/2249804666 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f4004160 con 0x7f00f8105e00 2023-07-14T18:16:53.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.120+0000 7f00fd18c700 1 -- 172.21.15.64:0/2249804666 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00f40042c0 con 0x7f00f8105e00 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.120+0000 7f00fd18c700 1 -- 172.21.15.64:0/2249804666 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f40118e0 con 0x7f00f8105e00 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 -- 172.21.15.64:0/2249804666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 msgr2=0x7f00f81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 --2- 172.21.15.64:0/2249804666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f00f40098a0 tx=0x7f00f4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 -- 172.21.15.64:0/2249804666 shutdown_connections 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 --2- 172.21.15.64:0/2249804666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f81081e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 -- 172.21.15.64:0/2249804666 >> 172.21.15.64:0/2249804666 conn(0x7f00f80ff840 msgr2=0x7f00f8101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 -- 172.21.15.64:0/2249804666 shutdown_connections 2023-07-14T18:16:53.219 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.121+0000 7f01003f2700 1 -- 172.21.15.64:0/2249804666 wait complete. 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f01003f2700 1 Processor -- start 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f01003f2700 1 -- start start 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f01003f2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f01003f2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f81995d0 con 0x7f00f8105e00 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f00fe18e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f00fe18e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45722/0 (socket says 172.21.15.64:45722) 2023-07-14T18:16:53.220 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f00fe18e700 1 -- 172.21.15.64:0/466348779 learned_addr learned my addr 172.21.15.64:0/466348779 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.122+0000 7f00fe18e700 1 -- 172.21.15.64:0/466348779 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f4009550 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f00fe18e700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00f4006af0 tx=0x7f00f4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f40037d0 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f00f401b440 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f81997d0 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00f4023400 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.123+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f8199bf0 con 0x7f00f8105e00 2023-07-14T18:16:53.221 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f00f4023670 con 0x7f00f8105e00 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f00ef7fe700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 0x7f00e403abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f00f404c6e0 con 0x7f00f8105e00 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f00fd98d700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 0x7f00e403abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f8061c70 con 0x7f00f8105e00 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.124+0000 7f00fd98d700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 0x7f00e403abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f00e8006fd0 tx=0x7f00e8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.129+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f00f4023920 con 0x7f00f8105e00 2023-07-14T18:16:53.222 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.521+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f00f819c630 con 0x7f00e4038710 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.526+0000 7f00ef7fe700 1 -- 172.21.15.64:0/466348779 <== mgr.14120 v2:172.21.15.64:6800/792928752 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f00f819c630 con 0x7f00e4038710 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 msgr2=0x7f00e403abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 0x7f00e403abc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f00e8006fd0 tx=0x7f00e8006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 msgr2=0x7f00f8199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f00f4006af0 tx=0x7f00f4011ed0 comp rx=0 tx=0).stop 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 shutdown_connections 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f00e4038710 0x7f00e403abc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:53.223 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 --2- 172.21.15.64:0/466348779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00f8105e00 0x7f00f8199090 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:53.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 >> 172.21.15.64:0/466348779 conn(0x7f00f80ff840 msgr2=0x7f00f8100f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:53.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 shutdown_connections 2023-07-14T18:16:53.224 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:52.528+0000 7f01003f2700 1 -- 172.21.15.64:0/466348779 wait complete. 2023-07-14T18:16:53.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:53 smithi064 ceph-mon[96720]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:16:53.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:53 smithi064 ceph-mon[96720]: Saving service mgr spec with placement count:2 2023-07-14T18:16:53.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:53 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:54.461 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.738+0000 7fbe06a29700 1 Processor -- start 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.738+0000 7fbe06a29700 1 -- start start 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbe06a29700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbe06a29700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe00106130 con 0x7fbe001057e0 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45726/0 (socket says 172.21.15.64:45726) 2023-07-14T18:16:54.462 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbdfffff700 1 -- 172.21.15.64:0/2712293148 learned_addr learned my addr 172.21.15.64:0/2712293148 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbdfffff700 1 -- 172.21.15.64:0/2712293148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00106270 con 0x7fbe001057e0 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.739+0000 7fbdfffff700 1 --2- 172.21.15.64:0/2712293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00098a0 tx=0x7fbdf0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5743c7b768cc01a7 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbdfeffd700 1 -- 172.21.15.64:0/2712293148 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdf0004160 con 0x7fbe001057e0 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbdfeffd700 1 -- 172.21.15.64:0/2712293148 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbdf00042c0 con 0x7fbe001057e0 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbdfeffd700 1 -- 172.21.15.64:0/2712293148 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdf00118e0 con 0x7fbe001057e0 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbe06a29700 1 -- 172.21.15.64:0/2712293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 msgr2=0x7fbe00105bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:54.463 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbe06a29700 1 --2- 172.21.15.64:0/2712293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbdf00098a0 tx=0x7fbdf0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbe06a29700 1 -- 172.21.15.64:0/2712293148 shutdown_connections 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbe06a29700 1 --2- 172.21.15.64:0/2712293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00105bf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.740+0000 7fbe06a29700 1 -- 172.21.15.64:0/2712293148 >> 172.21.15.64:0/2712293148 conn(0x7fbe000feff0 msgr2=0x7fbe00101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.741+0000 7fbe06a29700 1 -- 172.21.15.64:0/2712293148 shutdown_connections 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.741+0000 7fbe06a29700 1 -- 172.21.15.64:0/2712293148 wait complete. 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.741+0000 7fbe06a29700 1 Processor -- start 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.741+0000 7fbe06a29700 1 -- start start 2023-07-14T18:16:54.464 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbe06a29700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbe06a29700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0019be20 con 0x7fbe001057e0 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbdfffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45728/0 (socket says 172.21.15.64:45728) 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbdfffff700 1 -- 172.21.15.64:0/1391228192 learned_addr learned my addr 172.21.15.64:0/1391228192 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbdfffff700 1 -- 172.21.15.64:0/1391228192 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0009550 con 0x7fbe001057e0 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.742+0000 7fbdfffff700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006af0 tx=0x7fbdf0011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:54.465 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.743+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdf00037d0 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.743+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbdf001b440 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.743+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0019bf60 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.743+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbdf0023400 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.743+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0019c320 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.744+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fbdf00236c0 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.744+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe00191f90 con 0x7fbe001057e0 2023-07-14T18:16:54.466 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.744+0000 7fbdfd7fa700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 0x7fbde803ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.744+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbdf004c6f0 con 0x7fbe001057e0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.744+0000 7fbdff7fe700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 0x7fbde803ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.745+0000 7fbdff7fe700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 0x7fbde803ab70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4006fd0 tx=0x7fbdf4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:53.753+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbdf0023970 con 0x7fbe001057e0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.111+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fbe0002c9d0 con 0x7fbe001057e0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.114+0000 7fbdfd7fa700 1 -- 172.21.15.64:0/1391228192 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fbdf002bb40 con 0x7fbe001057e0 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 msgr2=0x7fbde803ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 0x7fbde803ab70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4006fd0 tx=0x7fbdf4006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 msgr2=0x7fbe00103f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0006af0 tx=0x7fbdf0011ed0 comp rx=0 tx=0).stop 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 shutdown_connections 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fbde80386c0 0x7fbde803ab70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 --2- 172.21.15.64:0/1391228192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe001057e0 0x7fbe00103f80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 >> 172.21.15.64:0/1391228192 conn(0x7fbe000feff0 msgr2=0x7fbe001013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 shutdown_connections 2023-07-14T18:16:54.468 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.117+0000 7fbe06a29700 1 -- 172.21.15.64:0/1391228192 wait complete. 2023-07-14T18:16:54.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:54 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:54.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:54 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1391228192' entity='client.admin' 2023-07-14T18:16:54.747 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:54 smithi064 ceph-mon[96720]: from='mgr.14120 172.21.15.64:0/2148104888' entity='mgr.x' 2023-07-14T18:16:55.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.952+0000 7f0efbb4e700 1 Processor -- start 2023-07-14T18:16:55.914 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.952+0000 7f0efbb4e700 1 -- start start 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0efbb4e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0efbb4e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4104b20 con 0x7f0ef41041d0 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef98ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef98ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45732/0 (socket says 172.21.15.64:45732) 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef98ea700 1 -- 172.21.15.64:0/1743749563 learned_addr learned my addr 172.21.15.64:0/1743749563 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef98ea700 1 -- 172.21.15.64:0/1743749563 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4104c60 con 0x7f0ef41041d0 2023-07-14T18:16:55.915 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef98ea700 1 --2- 172.21.15.64:0/1743749563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40098a0 tx=0x7f0ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3cad958c5de2457 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef88e8700 1 -- 172.21.15.64:0/1743749563 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee4004160 con 0x7f0ef41041d0 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef88e8700 1 -- 172.21.15.64:0/1743749563 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ee40042c0 con 0x7f0ef41041d0 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.953+0000 7f0ef88e8700 1 -- 172.21.15.64:0/1743749563 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee40118e0 con 0x7f0ef41041d0 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1743749563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 msgr2=0x7f0ef41045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1743749563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0ee40098a0 tx=0x7f0ee4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1743749563 shutdown_connections 2023-07-14T18:16:55.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1743749563 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef41045e0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1743749563 >> 172.21.15.64:0/1743749563 conn(0x7f0ef40ff840 msgr2=0x7f0ef4101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1743749563 shutdown_connections 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.954+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1743749563 wait complete. 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0efbb4e700 1 Processor -- start 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0efbb4e700 1 -- start start 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0efbb4e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0efbb4e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef4198470 con 0x7f0ef41041d0 2023-07-14T18:16:55.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0ef98ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0ef98ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45734/0 (socket says 172.21.15.64:45734) 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0ef98ea700 1 -- 172.21.15.64:0/1813402757 learned_addr learned my addr 172.21.15.64:0/1813402757 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.955+0000 7f0ef98ea700 1 -- 172.21.15.64:0/1813402757 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4009550 con 0x7f0ef41041d0 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0ef98ea700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4003e30 tx=0x7f0ee40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee40036a0 con 0x7f0ef41041d0 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ee401b440 con 0x7f0ef41041d0 2023-07-14T18:16:55.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef4198670 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ee4023400 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.956+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef4198a90 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.957+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f0ee4023630 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.957+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef4061c70 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.957+0000 7f0eeaffd700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 0x7f0ee003ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.957+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0ee404c800 con 0x7f0ef41041d0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.957+0000 7f0ef90e9700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 0x7f0ee003ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:55.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.958+0000 7f0ef90e9700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 0x7f0ee003ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0006fd0 tx=0x7f0ef0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:54.962+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ee40238e0 con 0x7f0ef41041d0 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.331+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0ef419b8f0 con 0x7f0ef41041d0 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.334+0000 7f0eeaffd700 1 -- 172.21.15.64:0/1813402757 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0ee402bb40 con 0x7f0ef41041d0 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.336+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 msgr2=0x7f0ee003ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.336+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 0x7f0ee003ac10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0006fd0 tx=0x7f0ef0006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.336+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 msgr2=0x7f0ef4197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:55.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.336+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4003e30 tx=0x7f0ee40119a0 comp rx=0 tx=0).stop 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 shutdown_connections 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f0ee0038760 0x7f0ee003ac10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 --2- 172.21.15.64:0/1813402757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ef41041d0 0x7f0ef4197f30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 >> 172.21.15.64:0/1813402757 conn(0x7f0ef40ff840 msgr2=0x7f0ef4100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 shutdown_connections 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:55.337+0000 7f0efbb4e700 1 -- 172.21.15.64:0/1813402757 wait complete. 2023-07-14T18:16:55.921 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling the dashboard module... 2023-07-14T18:16:56.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:56 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1813402757' entity='client.admin' 2023-07-14T18:16:57.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/894489388' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T18:16:57.685 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: ignoring --setuser ceph since I am not root 2023-07-14T18:16:57.685 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: ignoring --setgroup ceph since I am not root 2023-07-14T18:16:57.685 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:57.542+0000 7f955ffbf200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:16:57.685 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:57.638+0000 7f955ffbf200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:16:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa16592f700 1 Processor -- start 2023-07-14T18:16:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa16592f700 1 -- start start 2023-07-14T18:16:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa16592f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa16592f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa160105c80 con 0x7fa160073b40 2023-07-14T18:16:57.753 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa15effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.547+0000 7fa15effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45738/0 (socket says 172.21.15.64:45738) 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15effd700 1 -- 172.21.15.64:0/465836704 learned_addr learned my addr 172.21.15.64:0/465836704 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15effd700 1 -- 172.21.15.64:0/465836704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16010a7d0 con 0x7fa160073b40 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15effd700 1 --2- 172.21.15.64:0/465836704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1540098a0 tx=0x7fa154009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcfc6583c8eab34 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15dffb700 1 -- 172.21.15.64:0/465836704 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa154004160 con 0x7fa160073b40 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15dffb700 1 -- 172.21.15.64:0/465836704 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa1540042c0 con 0x7fa160073b40 2023-07-14T18:16:57.754 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.548+0000 7fa15dffb700 1 -- 172.21.15.64:0/465836704 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1540118e0 con 0x7fa160073b40 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 -- 172.21.15.64:0/465836704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 msgr2=0x7fa16010a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 --2- 172.21.15.64:0/465836704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fa1540098a0 tx=0x7fa154009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 -- 172.21.15.64:0/465836704 shutdown_connections 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 --2- 172.21.15.64:0/465836704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16010a290 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 -- 172.21.15.64:0/465836704 >> 172.21.15.64:0/465836704 conn(0x7fa1600ff840 msgr2=0x7fa160101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 -- 172.21.15.64:0/465836704 shutdown_connections 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 -- 172.21.15.64:0/465836704 wait complete. 2023-07-14T18:16:57.755 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.549+0000 7fa16592f700 1 Processor -- start 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16592f700 1 -- start start 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16592f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16592f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16019b7f0 con 0x7fa160073b40 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa15effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa15effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45740/0 (socket says 172.21.15.64:45740) 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa15effd700 1 -- 172.21.15.64:0/894489388 learned_addr learned my addr 172.21.15.64:0/894489388 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:57.756 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa15effd700 1 -- 172.21.15.64:0/894489388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa154009550 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa15effd700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa154003e30 tx=0x7fa154011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1540036a0 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa15401b440 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16019b9f0 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.550+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa1540193d0 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.551+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa16019be10 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa154019640 con 0x7fa160073b40 2023-07-14T18:16:57.757 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa16492d700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 0x7fa14803abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa15404c4d0 con 0x7fa160073b40 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa160195260 con 0x7fa160073b40 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa15e7fc700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 0x7fa14803abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.552+0000 7fa15e7fc700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 0x7fa14803abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa150006fd0 tx=0x7fa150006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:56.557+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa154018940 con 0x7fa160073b40 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.022+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa160061c70 con 0x7fa160073b40 2023-07-14T18:16:57.758 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.337+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fa15401b5b0 con 0x7fa160073b40 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.339+0000 7fa16492d700 1 -- 172.21.15.64:0/894489388 <== mon.0 v2:172.21.15.64:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa154016de0 con 0x7fa160073b40 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 msgr2=0x7fa14803abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 0x7fa14803abc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa150006fd0 tx=0x7fa150006e40 comp rx=0 tx=0).stop 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 msgr2=0x7fa16019b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa154003e30 tx=0x7fa154011e80 comp rx=0 tx=0).stop 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 shutdown_connections 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fa148038710 0x7fa14803abc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:57.759 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 --2- 172.21.15.64:0/894489388 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa160073b40 0x7fa16019b2b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:57.760 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.341+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 >> 172.21.15.64:0/894489388 conn(0x7fa1600ff840 msgr2=0x7fa160100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:57.760 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.342+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 shutdown_connections 2023-07-14T18:16:57.760 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:57.342+0000 7fa16592f700 1 -- 172.21.15.64:0/894489388 wait complete. 2023-07-14T18:16:57.954 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:57.752+0000 7f955ffbf200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:16:57.955 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:57.864+0000 7f955ffbf200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:16:58.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:58.078+0000 7f955ffbf200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:16:58.634 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:58.478+0000 7f955ffbf200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:16:58.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/894489388' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T18:16:58.634 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-mon[96720]: mgrmap e9: x(active, since 19s) 2023-07-14T18:16:58.915 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:58.588+0000 7f955ffbf200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:16:58.916 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:58.741+0000 7f955ffbf200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:16:59.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:16:59.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T18:16:59.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:16:59.103 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e99c5700 1 Processor -- start 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e99c5700 1 -- start start 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e99c5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e99c5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4105c40 con 0x7f87e41052f0 2023-07-14T18:16:59.104 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45754/0 (socket says 172.21.15.64:45754) 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.200+0000 7f87e2ffd700 1 -- 172.21.15.64:0/2102660835 learned_addr learned my addr 172.21.15.64:0/2102660835 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.201+0000 7f87e2ffd700 1 -- 172.21.15.64:0/2102660835 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4105d80 con 0x7f87e41052f0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.201+0000 7f87e2ffd700 1 --2- 172.21.15.64:0/2102660835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87d40098a0 tx=0x7f87d4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f45c066bbd4d06 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.201+0000 7f87e1ffb700 1 -- 172.21.15.64:0/2102660835 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d4004160 con 0x7f87e41052f0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.201+0000 7f87e1ffb700 1 -- 172.21.15.64:0/2102660835 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87d40042c0 con 0x7f87e41052f0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.201+0000 7f87e1ffb700 1 -- 172.21.15.64:0/2102660835 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d40118e0 con 0x7f87e41052f0 2023-07-14T18:16:59.105 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 -- 172.21.15.64:0/2102660835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 msgr2=0x7f87e4105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 --2- 172.21.15.64:0/2102660835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87d40098a0 tx=0x7f87d4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 -- 172.21.15.64:0/2102660835 shutdown_connections 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 --2- 172.21.15.64:0/2102660835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e4105700 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 -- 172.21.15.64:0/2102660835 >> 172.21.15.64:0/2102660835 conn(0x7f87e4100960 msgr2=0x7f87e4102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 -- 172.21.15.64:0/2102660835 shutdown_connections 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 -- 172.21.15.64:0/2102660835 wait complete. 2023-07-14T18:16:59.106 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.202+0000 7f87e99c5700 1 Processor -- start 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e99c5700 1 -- start start 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e99c5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e99c5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e4199620 con 0x7f87e41052f0 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45756/0 (socket says 172.21.15.64:45756) 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e2ffd700 1 -- 172.21.15.64:0/254644509 learned_addr learned my addr 172.21.15.64:0/254644509 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e2ffd700 1 -- 172.21.15.64:0/254644509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4009550 con 0x7f87e41052f0 2023-07-14T18:16:59.107 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e2ffd700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87d4003e30 tx=0x7f87d40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.203+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d40036a0 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.204+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f87d401b440 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.204+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f87d4023400 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.204+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e4199820 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.204+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e4199c40 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f87d401b5b0 con 0x7f87e41052f0 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e89c3700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 0x7f87cc03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:16:59.108 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f87d404c730 con 0x7f87e41052f0 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e27fc700 1 -- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 msgr2=0x7f87cc03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e27fc700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 0x7f87cc03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.205+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e41930b0 con 0x7f87e41052f0 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.210+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f87d4019840 con 0x7f87e41052f0 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.406+0000 7f87e27fc700 1 -- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 msgr2=0x7f87cc03ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:16:59.109 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.406+0000 7f87e27fc700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 0x7f87cc03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.679+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f87e4107230 con 0x7f87e41052f0 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.680+0000 7f87e89c3700 1 -- 172.21.15.64:0/254644509 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f87d4019840 con 0x7f87e41052f0 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 msgr2=0x7f87cc03ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 0x7f87cc03ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 msgr2=0x7f87e41990e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f87d4003e30 tx=0x7f87d40119a0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 shutdown_connections 2023-07-14T18:16:59.110 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7f87cc038760 0x7f87cc03ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 --2- 172.21.15.64:0/254644509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87e41052f0 0x7f87e41990e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 >> 172.21.15.64:0/254644509 conn(0x7f87e4100960 msgr2=0x7f87e41020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 shutdown_connections 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:58.682+0000 7f87e99c5700 1 -- 172.21.15.64:0/254644509 wait complete. 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for the mgr to restart... 2023-07-14T18:16:59.111 INFO:teuthology.orchestra.run.smithi064.stdout:Waiting for mgr epoch 9... 2023-07-14T18:16:59.296 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:58 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:58.869+0000 7f955ffbf200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:16:59.551 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:16:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:16:59.398+0000 7f955ffbf200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:16:59.835 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:16:59 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/254644509' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:17:00.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:00 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:00.083+0000 7f955ffbf200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:17:00.586 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:00 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:00.275+0000 7f955ffbf200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:17:01.585 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:01.239+0000 7f955ffbf200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:17:01.586 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:01.350+0000 7f955ffbf200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:17:01.963 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:01.569+0000 7f955ffbf200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:17:02.217 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:01.916+0000 7f955ffbf200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:17:02.586 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:02 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:02.171+0000 7f955ffbf200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:17:02.586 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:02 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:02.352+0000 7f955ffbf200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:17:03.086 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:02 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:02.680+0000 7f955ffbf200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:17:03.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:03 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:03.506+0000 7f955ffbf200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:17:03.836 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:03 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:03.664+0000 7f955ffbf200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:17:04.335 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:03 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:03.908+0000 7f955ffbf200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:17:04.336 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:04 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:04.046+0000 7f955ffbf200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:17:04.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:04 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:04.320+0000 7f955ffbf200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:17:05.836 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:17:05.366+0000 7f955ffbf200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:17:05.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: Active manager daemon x restarted 2023-07-14T18:17:05.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: Activating manager daemon x 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: mgrmap e10: x(active, starting, since 0.00305046s) 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:17:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:17:05.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: Manager daemon x is now available 2023-07-14T18:17:05.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:05.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:06.735 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: [14/Jul/2023:18:17:06] ENGINE Bus STARTING 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: [14/Jul/2023:18:17:06] ENGINE Serving on http://172.21.15.64:8765 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: [14/Jul/2023:18:17:06] ENGINE Serving on https://172.21.15.64:7150 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: [14/Jul/2023:18:17:06] ENGINE Bus STARTED 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:17:06.736 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: mgrmap e11: x(active, since 1.00571s) 2023-07-14T18:17:06.737 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:06 smithi064 ceph-mon[96720]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.507+0000 7fddb0396700 1 Processor -- start 2023-07-14T18:17:06.765 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.507+0000 7fddb0396700 1 -- start start 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddb0396700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddb0396700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8107310 con 0x7fdda81069c0 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddae132700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddae132700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45764/0 (socket says 172.21.15.64:45764) 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddae132700 1 -- 172.21.15.64:0/1369091114 learned_addr learned my addr 172.21.15.64:0/1369091114 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.508+0000 7fddae132700 1 -- 172.21.15.64:0/1369091114 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda8107450 con 0x7fdda81069c0 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddae132700 1 --2- 172.21.15.64:0/1369091114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdda40098a0 tx=0x7fdda4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a81f48a09b4a0ea server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:06.766 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddad130700 1 -- 172.21.15.64:0/1369091114 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdda4004160 con 0x7fdda81069c0 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddad130700 1 -- 172.21.15.64:0/1369091114 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdda40042c0 con 0x7fdda81069c0 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddad130700 1 -- 172.21.15.64:0/1369091114 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdda40118e0 con 0x7fdda81069c0 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddb0396700 1 -- 172.21.15.64:0/1369091114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 msgr2=0x7fdda8106dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.509+0000 7fddb0396700 1 --2- 172.21.15.64:0/1369091114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdda40098a0 tx=0x7fdda4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 -- 172.21.15.64:0/1369091114 shutdown_connections 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 --2- 172.21.15.64:0/1369091114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda8106dd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.767 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 -- 172.21.15.64:0/1369091114 >> 172.21.15.64:0/1369091114 conn(0x7fdda80feff0 msgr2=0x7fdda8101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 -- 172.21.15.64:0/1369091114 shutdown_connections 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 -- 172.21.15.64:0/1369091114 wait complete. 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 Processor -- start 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.510+0000 7fddb0396700 1 -- start start 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddb0396700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddb0396700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdda8106600 con 0x7fdda81069c0 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddae132700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:06.768 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddae132700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45766/0 (socket says 172.21.15.64:45766) 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddae132700 1 -- 172.21.15.64:0/2140329152 learned_addr learned my addr 172.21.15.64:0/2140329152 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddae132700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda4009550 con 0x7fdda81069c0 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddae132700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdda4003e30 tx=0x7fdda40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdda40036a0 con 0x7fdda81069c0 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda8104710 con 0x7fdda81069c0 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdda401b440 con 0x7fdda81069c0 2023-07-14T18:17:06.769 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdda4023400 con 0x7fdda81069c0 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.511+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda8104b30 con 0x7fdda81069c0 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.512+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fdda401b5b0 con 0x7fdda81069c0 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.513+0000 7fdd9f7fe700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.513+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd9403b320 con 0x7fdd94038760 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.513+0000 7fddad931700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.513+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdda404c700 con 0x7fdda81069c0 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.513+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:17:06.770 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.714+0000 7fddad931700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:16:59.714+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:00.115+0000 7fddad931700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:00.115+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:00.916+0000 7fddad931700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:00.916+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:02.517+0000 7fddad931700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.64:6800/792928752 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:02.517+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:17:06.771 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:05.381+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7fdda4023680 con 0x7fdda81069c0 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:05.381+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 msgr2=0x7fdd9403ac10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:05.381+0000 7fdd9f7fe700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.382+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fdda401dda0 con 0x7fdda81069c0 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.382+0000 7fdd9f7fe700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 0x7fdd9403e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.382+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd9403b320 con 0x7fdd9403bff0 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.383+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 0x7fdd9403e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:06.772 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.383+0000 7fddad931700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 0x7fdd9403e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdd98003a10 tx=0x7fdd980092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.384+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fdd9403b320 con 0x7fdd9403bff0 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.394+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdda8105c60 con 0x7fdd9403bff0 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fdd9f7fe700 1 -- 172.21.15.64:0/2140329152 <== mgr.14152 v2:172.21.15.64:6800/387190897 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fdda8105c60 con 0x7fdd9403bff0 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 msgr2=0x7fdd9403e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 0x7fdd9403e3d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdd98003a10 tx=0x7fdd980092b0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 msgr2=0x7fdda81060c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdda4003e30 tx=0x7fdda40119a0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.773 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 shutdown_connections 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd9403bff0 0x7fdd9403e3d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdda81069c0 0x7fdda81060c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 --2- 172.21.15.64:0/2140329152 >> [v2:172.21.15.64:6800/792928752,v1:172.21.15.64:6801/792928752] conn(0x7fdd94038760 0x7fdd9403ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 >> 172.21.15.64:0/2140329152 conn(0x7fdda80feff0 msgr2=0x7fdda8100760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 shutdown_connections 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:06.395+0000 7fddb0396700 1 -- 172.21.15.64:0/2140329152 wait complete. 2023-07-14T18:17:06.774 INFO:teuthology.orchestra.run.smithi064.stdout:mgr epoch 9 is available 2023-07-14T18:17:06.775 INFO:teuthology.orchestra.run.smithi064.stdout:Generating a dashboard self-signed certificate... 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.305+0000 7fe373ed8700 1 Processor -- start 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.305+0000 7fe373ed8700 1 -- start start 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe373ed8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe373ed8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c103f40 con 0x7fe36c1035f0 2023-07-14T18:17:08.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe371c74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe371c74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45820/0 (socket says 172.21.15.64:45820) 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe371c74700 1 -- 172.21.15.64:0/4038216810 learned_addr learned my addr 172.21.15.64:0/4038216810 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe371c74700 1 -- 172.21.15.64:0/4038216810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c104080 con 0x7fe36c1035f0 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.306+0000 7fe371c74700 1 --2- 172.21.15.64:0/4038216810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3680098a0 tx=0x7fe368009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dcefd01094f545a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe370c72700 1 -- 172.21.15.64:0/4038216810 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe368004160 con 0x7fe36c1035f0 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe370c72700 1 -- 172.21.15.64:0/4038216810 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe3680042c0 con 0x7fe36c1035f0 2023-07-14T18:17:08.212 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe370c72700 1 -- 172.21.15.64:0/4038216810 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3680118e0 con 0x7fe36c1035f0 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe373ed8700 1 -- 172.21.15.64:0/4038216810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 msgr2=0x7fe36c103a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe373ed8700 1 --2- 172.21.15.64:0/4038216810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe3680098a0 tx=0x7fe368009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe373ed8700 1 -- 172.21.15.64:0/4038216810 shutdown_connections 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.307+0000 7fe373ed8700 1 --2- 172.21.15.64:0/4038216810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c103a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 -- 172.21.15.64:0/4038216810 >> 172.21.15.64:0/4038216810 conn(0x7fe36c0feff0 msgr2=0x7fe36c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 -- 172.21.15.64:0/4038216810 shutdown_connections 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 -- 172.21.15.64:0/4038216810 wait complete. 2023-07-14T18:17:08.213 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 Processor -- start 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 -- start start 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.308+0000 7fe373ed8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c197420 con 0x7fe36c1035f0 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe371c74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe371c74700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45822/0 (socket says 172.21.15.64:45822) 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe371c74700 1 -- 172.21.15.64:0/1554271629 learned_addr learned my addr 172.21.15.64:0/1554271629 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:08.214 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe371c74700 1 -- 172.21.15.64:0/1554271629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe368009550 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe371c74700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe368006af0 tx=0x7fe368011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3680037d0 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe36801b440 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c197620 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe368023400 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.309+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c197a40 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.310+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fe3680236c0 con 0x7fe36c1035f0 2023-07-14T18:17:08.215 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.311+0000 7fe362ffd700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 0x7fe35803eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.311+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe36804c6a0 con 0x7fe36c1035f0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.311+0000 7fe371473700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 0x7fe35803eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.311+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c190d10 con 0x7fe36c1035f0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.311+0000 7fe371473700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 0x7fe35803eea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006fd0 tx=0x7fe35c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.316+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe3680239c0 con 0x7fe36c1035f0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.686+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe36c060a80 con 0x7fe35803c9f0 2023-07-14T18:17:08.216 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.818+0000 7fe362ffd700 1 -- 172.21.15.64:0/1554271629 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe36c060a80 con 0x7fe35803c9f0 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 msgr2=0x7fe35803eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 0x7fe35803eea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe35c006fd0 tx=0x7fe35c006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 msgr2=0x7fe36c196ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe368006af0 tx=0x7fe368011ed0 comp rx=0 tx=0).stop 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 shutdown_connections 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe35803c9f0 0x7fe35803eea0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 --2- 172.21.15.64:0/1554271629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe36c1035f0 0x7fe36c196ee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:08.217 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 >> 172.21.15.64:0/1554271629 conn(0x7fe36c0feff0 msgr2=0x7fe36c100920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:08.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 shutdown_connections 2023-07-14T18:17:08.218 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:07.821+0000 7fe373ed8700 1 -- 172.21.15.64:0/1554271629 wait complete. 2023-07-14T18:17:08.218 INFO:teuthology.orchestra.run.smithi064.stdout:Creating initial admin user... 2023-07-14T18:17:08.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:08.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:08.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:09.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:09 smithi064 ceph-mon[96720]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:09.531 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:09 smithi064 ceph-mon[96720]: mgrmap e12: x(active, since 2s) 2023-07-14T18:17:09.531 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:10.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$MriL4yhPUUfSGr9MeB/dQOfomOYN58OklkMVzGvR9bvx0KBof7QBK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689358629, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T18:17:10.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8c946700 1 Processor -- start 2023-07-14T18:17:10.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8c946700 1 -- start start 2023-07-14T18:17:10.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8c946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:10.058 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8c946700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf881083f0 con 0x7fcf88073b60 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45826/0 (socket says 172.21.15.64:45826) 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.621+0000 7fcf8659c700 1 -- 172.21.15.64:0/2332977159 learned_addr learned my addr 172.21.15.64:0/2332977159 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.622+0000 7fcf8659c700 1 -- 172.21.15.64:0/2332977159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88108530 con 0x7fcf88073b60 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.622+0000 7fcf8659c700 1 --2- 172.21.15.64:0/2332977159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf780098a0 tx=0x7fcf78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da1dd2464f3c4c1a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.622+0000 7fcf8559a700 1 -- 172.21.15.64:0/2332977159 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf78004160 con 0x7fcf88073b60 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.622+0000 7fcf8559a700 1 -- 172.21.15.64:0/2332977159 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf780042c0 con 0x7fcf88073b60 2023-07-14T18:17:10.059 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.622+0000 7fcf8559a700 1 -- 172.21.15.64:0/2332977159 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf780118e0 con 0x7fcf88073b60 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 -- 172.21.15.64:0/2332977159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 msgr2=0x7fcf88107eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2332977159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcf780098a0 tx=0x7fcf78009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 -- 172.21.15.64:0/2332977159 shutdown_connections 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2332977159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf88073b60 0x7fcf88107eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 -- 172.21.15.64:0/2332977159 >> 172.21.15.64:0/2332977159 conn(0x7fcf880ff880 msgr2=0x7fcf88101c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 -- 172.21.15.64:0/2332977159 shutdown_connections 2023-07-14T18:17:10.060 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.623+0000 7fcf8c946700 1 -- 172.21.15.64:0/2332977159 wait complete. 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8c946700 1 Processor -- start 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8c946700 1 -- start start 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8c946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8c946700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf88105bd0 con 0x7fcf881057c0 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45828/0 (socket says 172.21.15.64:45828) 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8659c700 1 -- 172.21.15.64:0/2837628534 learned_addr learned my addr 172.21.15.64:0/2837628534 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:10.061 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8659c700 1 -- 172.21.15.64:0/2837628534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf78009550 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.624+0000 7fcf8659c700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcf78003e30 tx=0x7fcf780119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.625+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf780036a0 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.625+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf7801b440 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.625+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf88104390 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.625+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf78023400 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.625+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf881047b0 con 0x7fcf881057c0 2023-07-14T18:17:10.062 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fcf7801b5b0 con 0x7fcf881057c0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf777fe700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 0x7fcf7003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcf7804c780 con 0x7fcf881057c0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf85d9b700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 0x7fcf7003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf881930a0 con 0x7fcf881057c0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.626+0000 7fcf85d9b700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 0x7fcf7003ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c006fd0 tx=0x7fcf7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:08.631+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf7801ba90 con 0x7fcf881057c0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.028+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcf88061a90 con 0x7fcf700387b0 2023-07-14T18:17:10.063 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.276+0000 7fcf777fe700 1 -- 172.21.15.64:0/2837628534 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fcf88061a90 con 0x7fcf700387b0 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 msgr2=0x7fcf7003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 0x7fcf7003ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c006fd0 tx=0x7fcf7c006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 msgr2=0x7fcf88103e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcf78003e30 tx=0x7fcf780119a0 comp rx=0 tx=0).stop 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 shutdown_connections 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcf700387b0 0x7fcf7003ac60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:10.064 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 --2- 172.21.15.64:0/2837628534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf881057c0 0x7fcf88103e50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:10.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 >> 172.21.15.64:0/2837628534 conn(0x7fcf880ff880 msgr2=0x7fcf88101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:10.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 shutdown_connections 2023-07-14T18:17:10.065 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:09.279+0000 7fcf8c946700 1 -- 172.21.15.64:0/2837628534 wait complete. 2023-07-14T18:17:10.065 INFO:teuthology.orchestra.run.smithi064.stdout:Fetching dashboard port number... 2023-07-14T18:17:10.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:10 smithi064 ceph-mon[96720]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:10.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:11.204 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.445+0000 7f0706a84700 1 Processor -- start 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.445+0000 7f0706a84700 1 -- start start 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0706a84700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0706a84700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700073be0 con 0x7f0700105de0 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0704820700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:11.205 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0704820700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45832/0 (socket says 172.21.15.64:45832) 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0704820700 1 -- 172.21.15.64:0/900948698 learned_addr learned my addr 172.21.15.64:0/900948698 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0704820700 1 -- 172.21.15.64:0/900948698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0700073d20 con 0x7f0700105de0 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f0704820700 1 --2- 172.21.15.64:0/900948698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06f40098a0 tx=0x7f06f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=630e4e3d742dc421 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f06ff7fe700 1 -- 172.21.15.64:0/900948698 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f4004160 con 0x7f0700105de0 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f06ff7fe700 1 -- 172.21.15.64:0/900948698 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f06f40042c0 con 0x7f0700105de0 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.446+0000 7f06ff7fe700 1 -- 172.21.15.64:0/900948698 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f40118e0 con 0x7f0700105de0 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 -- 172.21.15.64:0/900948698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 msgr2=0x7f07001081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:11.206 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 --2- 172.21.15.64:0/900948698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f06f40098a0 tx=0x7f06f4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 -- 172.21.15.64:0/900948698 shutdown_connections 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 --2- 172.21.15.64:0/900948698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f07001081c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 -- 172.21.15.64:0/900948698 >> 172.21.15.64:0/900948698 conn(0x7f07000ff840 msgr2=0x7f0700101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 -- 172.21.15.64:0/900948698 shutdown_connections 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.447+0000 7f0706a84700 1 -- 172.21.15.64:0/900948698 wait complete. 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0706a84700 1 Processor -- start 2023-07-14T18:17:11.207 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0706a84700 1 -- start start 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0706a84700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0706a84700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001995b0 con 0x7f0700105de0 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0704820700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0704820700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45834/0 (socket says 172.21.15.64:45834) 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0704820700 1 -- 172.21.15.64:0/881792224 learned_addr learned my addr 172.21.15.64:0/881792224 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.448+0000 7f0704820700 1 -- 172.21.15.64:0/881792224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f4009550 con 0x7f0700105de0 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f0704820700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06f4006af0 tx=0x7f06f4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:11.208 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f40037d0 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f06f401b440 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07001997b0 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f06f4023400 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.449+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0700199bd0 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.450+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f06f40236c0 con 0x7f0700105de0 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.450+0000 7f06fdffb700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 0x7f06f003abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:11.209 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.450+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f06f404d750 con 0x7f0700105de0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.450+0000 7f06fffff700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 0x7f06f003abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.450+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0700061c70 con 0x7f0700105de0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.451+0000 7f06fffff700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 0x7f06f003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06ec006fd0 tx=0x7f06ec006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.457+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f06f4018350 con 0x7f0700105de0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.820+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f070019c610 con 0x7f0700105de0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.820+0000 7f06fdffb700 1 -- 172.21.15.64:0/881792224 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f06f402bb40 con 0x7f0700105de0 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 msgr2=0x7f06f003abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:11.210 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 0x7f06f003abc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f06ec006fd0 tx=0x7f06ec006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 msgr2=0x7f0700199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06f4006af0 tx=0x7f06f4011ed0 comp rx=0 tx=0).stop 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 shutdown_connections 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f06f0038710 0x7f06f003abc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 --2- 172.21.15.64:0/881792224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700105de0 0x7f0700199070 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 >> 172.21.15.64:0/881792224 conn(0x7f07000ff840 msgr2=0x7f0700100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 shutdown_connections 2023-07-14T18:17:11.211 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:10.823+0000 7f0706a84700 1 -- 172.21.15.64:0/881792224 wait complete. 2023-07-14T18:17:11.214 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:17:11.214 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout disabled 2023-07-14T18:17:11.221 INFO:teuthology.orchestra.run.smithi064.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:17:11.221 INFO:teuthology.orchestra.run.smithi064.stdout:systemctl: stdout inactive 2023-07-14T18:17:11.222 INFO:teuthology.orchestra.run.smithi064.stdout:firewalld.service is not enabled 2023-07-14T18:17:11.222 INFO:teuthology.orchestra.run.smithi064.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T18:17:11.223 INFO:teuthology.orchestra.run.smithi064.stdout:Ceph Dashboard is now available at: 2023-07-14T18:17:11.223 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:11.223 INFO:teuthology.orchestra.run.smithi064.stdout: URL: https://smithi064:8443/ 2023-07-14T18:17:11.223 INFO:teuthology.orchestra.run.smithi064.stdout: User: admin 2023-07-14T18:17:11.224 INFO:teuthology.orchestra.run.smithi064.stdout: Password: mnpojb38rb 2023-07-14T18:17:11.224 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:11.224 INFO:teuthology.orchestra.run.smithi064.stdout:Saving cluster configuration to /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config directory 2023-07-14T18:17:11.224 INFO:teuthology.orchestra.run.smithi064.stdout:Enabling autotune for osd_memory_target 2023-07-14T18:17:11.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:11 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/881792224' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T18:17:12.470 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c9a18c700 1 Processor -- start 2023-07-14T18:17:12.470 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c9a18c700 1 -- start start 2023-07-14T18:17:12.470 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c9a18c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c9a18c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940796e0 con 0x7f0c9407aca0 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c937fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c937fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45842/0 (socket says 172.21.15.64:45842) 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.672+0000 7f0c937fe700 1 -- 172.21.15.64:0/2653935166 learned_addr learned my addr 172.21.15.64:0/2653935166 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.673+0000 7f0c937fe700 1 -- 172.21.15.64:0/2653935166 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94079820 con 0x7f0c9407aca0 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.673+0000 7f0c937fe700 1 --2- 172.21.15.64:0/2653935166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c880098a0 tx=0x7f0c88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8af982d1da4c4d2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:12.471 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.673+0000 7f0c927fc700 1 -- 172.21.15.64:0/2653935166 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c88004160 con 0x7f0c9407aca0 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.673+0000 7f0c927fc700 1 -- 172.21.15.64:0/2653935166 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c880042c0 con 0x7f0c9407aca0 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.673+0000 7f0c927fc700 1 -- 172.21.15.64:0/2653935166 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c880118e0 con 0x7f0c9407aca0 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2653935166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 msgr2=0x7f0c940791a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2653935166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0c880098a0 tx=0x7f0c88009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2653935166 shutdown_connections 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2653935166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c940791a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2653935166 >> 172.21.15.64:0/2653935166 conn(0x7f0c94100920 msgr2=0x7f0c94102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2653935166 shutdown_connections 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.674+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2653935166 wait complete. 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c9a18c700 1 Processor -- start 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c9a18c700 1 -- start start 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c9a18c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c9a18c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9419b7d0 con 0x7f0c9407aca0 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c937fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c937fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45844/0 (socket says 172.21.15.64:45844) 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c937fe700 1 -- 172.21.15.64:0/2681153298 learned_addr learned my addr 172.21.15.64:0/2681153298 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.675+0000 7f0c937fe700 1 -- 172.21.15.64:0/2681153298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c88009550 con 0x7f0c9407aca0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c937fe700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c88006af0 tx=0x7f0c88011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c880037d0 con 0x7f0c9407aca0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c8801b440 con 0x7f0c9407aca0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9419b9d0 con 0x7f0c9407aca0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0c88023400 con 0x7f0c9407aca0 2023-07-14T18:17:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.676+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9419bdf0 con 0x7f0c9407aca0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.677+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f0c88003930 con 0x7f0c9407aca0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.677+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c94195260 con 0x7f0c9407aca0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.677+0000 7f0c90ff9700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 0x7f0c7c03ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.677+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0c8804c7d0 con 0x7f0c9407aca0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.677+0000 7f0c92ffd700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 0x7f0c7c03ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.678+0000 7f0c92ffd700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 0x7f0c7c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006fd0 tx=0x7f0c84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:11.682+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c8802b350 con 0x7f0c9407aca0 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.034+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f0c9402cb60 con 0x7f0c9407aca0 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.034+0000 7f0c90ff9700 1 -- 172.21.15.64:0/2681153298 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0c8802b350 con 0x7f0c9407aca0 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 msgr2=0x7f0c7c03ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 0x7f0c7c03ac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0c84006fd0 tx=0x7f0c84006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 msgr2=0x7f0c9419b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0c88006af0 tx=0x7f0c88011ed0 comp rx=0 tx=0).stop 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 shutdown_connections 2023-07-14T18:17:12.476 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0c7c038760 0x7f0c7c03ac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:12.477 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 --2- 172.21.15.64:0/2681153298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c9407aca0 0x7f0c9419b290 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:12.477 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 >> 172.21.15.64:0/2681153298 conn(0x7f0c94100920 msgr2=0x7f0c94102040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:12.477 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 shutdown_connections 2023-07-14T18:17:12.477 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.036+0000 7f0c9a18c700 1 -- 172.21.15.64:0/2681153298 wait complete. 2023-07-14T18:17:13.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:13 smithi064 ceph-mon[96720]: mgrmap e13: x(active, since 6s) 2023-07-14T18:17:13.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8cb341700 1 Processor -- start 2023-07-14T18:17:13.916 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8cb341700 1 -- start start 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8cb341700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8cb341700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4073be0 con 0x7fa8c4105de0 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8c90dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8c90dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45848/0 (socket says 172.21.15.64:45848) 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.890+0000 7fa8c90dd700 1 -- 172.21.15.64:0/1011233251 learned_addr learned my addr 172.21.15.64:0/1011233251 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8c90dd700 1 -- 172.21.15.64:0/1011233251 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4073d20 con 0x7fa8c4105de0 2023-07-14T18:17:13.917 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8c90dd700 1 --2- 172.21.15.64:0/1011233251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00098a0 tx=0x7fa8c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2feba6b7710033 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8bbfff700 1 -- 172.21.15.64:0/1011233251 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8c0004160 con 0x7fa8c4105de0 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8bbfff700 1 -- 172.21.15.64:0/1011233251 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa8c00042c0 con 0x7fa8c4105de0 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8bbfff700 1 -- 172.21.15.64:0/1011233251 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8c00118e0 con 0x7fa8c4105de0 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8cb341700 1 -- 172.21.15.64:0/1011233251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 msgr2=0x7fa8c41081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.891+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1011233251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa8c00098a0 tx=0x7fa8c0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 -- 172.21.15.64:0/1011233251 shutdown_connections 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1011233251 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c41081c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:13.918 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 -- 172.21.15.64:0/1011233251 >> 172.21.15.64:0/1011233251 conn(0x7fa8c40ff840 msgr2=0x7fa8c4101c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 -- 172.21.15.64:0/1011233251 shutdown_connections 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 -- 172.21.15.64:0/1011233251 wait complete. 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.892+0000 7fa8cb341700 1 Processor -- start 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8cb341700 1 -- start start 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8cb341700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8cb341700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c4105c80 con 0x7fa8c4105de0 2023-07-14T18:17:13.919 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8c90dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8c90dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45850/0 (socket says 172.21.15.64:45850) 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8c90dd700 1 -- 172.21.15.64:0/1841921211 learned_addr learned my addr 172.21.15.64:0/1841921211 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8c90dd700 1 -- 172.21.15.64:0/1841921211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009550 con 0x7fa8c4105de0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.893+0000 7fa8c90dd700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006af0 tx=0x7fa8c0011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8c00037d0 con 0x7fa8c4105de0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa8c001b440 con 0x7fa8c4105de0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c4103d90 con 0x7fa8c4105de0 2023-07-14T18:17:13.920 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8c0023400 con 0x7fa8c4105de0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c41041b0 con 0x7fa8c4105de0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.894+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa8c00236c0 con 0x7fa8c4105de0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.895+0000 7fa8ba7fc700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 0x7fa8b003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.895+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa8c004cac0 con 0x7fa8c4105de0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.895+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c41093b0 con 0x7fa8c4105de0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.895+0000 7fa8c88dc700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 0x7fa8b003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:13.921 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.895+0000 7fa8c88dc700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 0x7fa8b003ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4006fd0 tx=0x7fa8b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:12.900+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8c00239c0 con 0x7fa8c4105de0 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.477+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa8c41095d0 con 0x7fa8c4105de0 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.480+0000 7fa8ba7fc700 1 -- 172.21.15.64:0/1841921211 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa8c002bb40 con 0x7fa8c4105de0 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 msgr2=0x7fa8b003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 0x7fa8b003ac60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4006fd0 tx=0x7fa8b4006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 msgr2=0x7fa8c4105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0006af0 tx=0x7fa8c0011ed0 comp rx=0 tx=0).stop 2023-07-14T18:17:13.922 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 shutdown_connections 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa8b00387b0 0x7fa8b003ac60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 --2- 172.21.15.64:0/1841921211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8c4105de0 0x7fa8c4105740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 >> 172.21.15.64:0/1841921211 conn(0x7fa8c40ff840 msgr2=0x7fa8c4100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 shutdown_connections 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr 2023-07-14T18:17:13.482+0000 7fa8cb341700 1 -- 172.21.15.64:0/1841921211 wait complete. 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T18:17:13.923 INFO:teuthology.orchestra.run.smithi064.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.924 INFO:teuthology.orchestra.run.smithi064.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: ceph telemetry on 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout:For more information see: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:13.925 INFO:teuthology.orchestra.run.smithi064.stdout:Bootstrap complete. 2023-07-14T18:17:13.950 INFO:tasks.cephadm:Fetching config... 2023-07-14T18:17:13.950 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:17:13.951 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T18:17:13.967 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T18:17:13.968 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:17:13.968 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T18:17:14.026 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T18:17:14.026 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:17:14.026 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T18:17:14.094 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T18:17:14.095 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:17:14.095 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T18:17:14.154 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T18:17:14.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:17:14.241 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:14.257 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:17:14.302 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:14.314 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:17:14.359 INFO:teuthology.orchestra.run.smithi144.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYiTilFAc28yxfFHuKiYFIGMyL3n6dsm0LoOA0kAUbO/MN0YcUfIx5ZsREll8/mIG1QZwCan1E4ZpQlS3nJvlTBvf3hgHUCugE0SHT4vW193woB4NA6mpJrub4x5Dc22u9JYNIQhBSodXpultmqWz8jQntYbWS6Pe+JNd0pFdZrv+DQduaV2IaCpW4DOQpeAw47Q4zc/pzl8JlQlM+TOWe60a8qxz3gJiNqaM/hxW85Kfdfn1pWgqAi249EmJ5iKSdDJX0LILuiT/LVQyFtYDJSUSTBab5M4yUlzdsYXdpiGY+VIYxnRGho4duU3IXchd8m/63NvlfoTlJS7vMDItocjR6TDVppnWNFYjsTmap9jYe6pmnhIAnnysIT3sSryHgB16AF/Q2H/VipU7D9X9A6AupkyvUvxDcZd6kfntyNsyOlTIuJPe8qrYh0jPpK8WiDf8qtKmQOVGTeBdxYU5bQaIl3ER0vyIaxfLSrBApnzzZCC7NxSSHzGBNspLSuC0= ceph-5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:14.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T18:17:14.821 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:14 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1841921211' entity='client.admin' 2023-07-14T18:17:14.835 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:16.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 -- 172.21.15.64:0/2633047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 msgr2=0x7fc8d4102580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2633047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d4102580 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009910 tx=0x7fc8c4009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 -- 172.21.15.64:0/2633047848 shutdown_connections 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2633047848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d4102580 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 -- 172.21.15.64:0/2633047848 >> 172.21.15.64:0/2633047848 conn(0x7fc8d40f9ba0 msgr2=0x7fc8d40fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 -- 172.21.15.64:0/2633047848 shutdown_connections 2023-07-14T18:17:16.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.167+0000 7fc8d8915700 1 -- 172.21.15.64:0/2633047848 wait complete. 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d8915700 1 Processor -- start 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d8915700 1 -- start start 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d8915700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d8915700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d4100040 con 0x7fc8d41001a0 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45856/0 (socket says 172.21.15.64:45856) 2023-07-14T18:17:16.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.168+0000 7fc8d259c700 1 -- 172.21.15.64:0/2596090934 learned_addr learned my addr 172.21.15.64:0/2596090934 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:16.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8d259c700 1 -- 172.21.15.64:0/2596090934 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c40095f0 con 0x7fc8d41001a0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8d259c700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4005760 tx=0x7fc8c4004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c400ba90 con 0x7fc8d41001a0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d40fe150 con 0x7fc8d41001a0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc8c400bbf0 con 0x7fc8d41001a0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc8c4017440 con 0x7fc8d41001a0 2023-07-14T18:17:16.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.169+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d40fe5f0 con 0x7fc8d41001a0 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.170+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d4061c70 con 0x7fc8d41001a0 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.170+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc8c400f460 con 0x7fc8d41001a0 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.170+0000 7fc8c37fe700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 0x7fc8bc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.170+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc8c404ea10 con 0x7fc8d41001a0 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.171+0000 7fc8d1d9b700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 0x7fc8bc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:16.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.171+0000 7fc8d1d9b700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 0x7fc8bc03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006fd0 tx=0x7fc8c8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:16.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.175+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc8c4015d20 con 0x7fc8d41001a0 2023-07-14T18:17:16.334 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:16.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:16.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:16.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:16.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:16.335 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:16.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.393+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fc8d418dd80 con 0x7fc8d41001a0 2023-07-14T18:17:16.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.396+0000 7fc8c37fe700 1 -- 172.21.15.64:0/2596090934 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc8c4027070 con 0x7fc8d41001a0 2023-07-14T18:17:16.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 msgr2=0x7fc8bc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:16.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 0x7fc8bc03ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006fd0 tx=0x7fc8c8006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:16.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 msgr2=0x7fc8d40ffb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:16.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4005760 tx=0x7fc8c4004f20 comp rx=0 tx=0).stop 2023-07-14T18:17:16.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 shutdown_connections 2023-07-14T18:17:16.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8bc0387b0 0x7fc8bc03ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:16.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.399+0000 7fc8d8915700 1 --2- 172.21.15.64:0/2596090934 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8d41001a0 0x7fc8d40ffb00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:16.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.400+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 >> 172.21.15.64:0/2596090934 conn(0x7fc8d40f9ba0 msgr2=0x7fc8d40fa930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:16.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.400+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 shutdown_connections 2023-07-14T18:17:16.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:16.400+0000 7fc8d8915700 1 -- 172.21.15.64:0/2596090934 wait complete. 2023-07-14T18:17:16.933 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T18:17:16.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T18:17:17.494 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:17.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:17 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2596090934' entity='client.admin' 2023-07-14T18:17:17.703 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:17.704 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 -- 172.21.15.64:0/3643227012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 msgr2=0x7fad800fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 --2- 172.21.15.64:0/3643227012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad800fea20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fad70009910 tx=0x7fad70009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 -- 172.21.15.64:0/3643227012 shutdown_connections 2023-07-14T18:17:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 --2- 172.21.15.64:0/3643227012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad800fea20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:18.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 -- 172.21.15.64:0/3643227012 >> 172.21.15.64:0/3643227012 conn(0x7fad800f9ba0 msgr2=0x7fad800fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 -- 172.21.15.64:0/3643227012 shutdown_connections 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.631+0000 7fad85216700 1 -- 172.21.15.64:0/3643227012 wait complete. 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.632+0000 7fad85216700 1 Processor -- start 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.632+0000 7fad85216700 1 -- start start 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.632+0000 7fad85216700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.632+0000 7fad85216700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad801974b0 con 0x7fad800fe610 2023-07-14T18:17:18.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad7ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:18.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad7ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45862/0 (socket says 172.21.15.64:45862) 2023-07-14T18:17:18.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad7ed9d700 1 -- 172.21.15.64:0/3692085359 learned_addr learned my addr 172.21.15.64:0/3692085359 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:18.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad7ed9d700 1 -- 172.21.15.64:0/3692085359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700095f0 con 0x7fad800fe610 2023-07-14T18:17:18.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad7ed9d700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fad70005950 tx=0x7fad70005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:18.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad7000ba90 con 0x7fad800fe610 2023-07-14T18:17:18.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fad7000bbf0 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad801975f0 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fad70017440 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.633+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad80197a90 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.634+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fad70017700 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.634+0000 7fad67fff700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 0x7fad6803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.634+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8018d090 con 0x7fad800fe610 2023-07-14T18:17:18.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.634+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fad7004ccc0 con 0x7fad800fe610 2023-07-14T18:17:18.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.635+0000 7fad7e59c700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 0x7fad6803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:18.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.635+0000 7fad7e59c700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 0x7fad6803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad74006fd0 tx=0x7fad74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:18.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.639+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad7002c440 con 0x7fad800fe610 2023-07-14T18:17:18.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.842+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fad80060b10 con 0x7fad680387b0 2023-07-14T18:17:18.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.846+0000 7fad67fff700 1 -- 172.21.15.64:0/3692085359 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad80060b10 con 0x7fad680387b0 2023-07-14T18:17:18.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 msgr2=0x7fad6803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 0x7fad6803ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fad74006fd0 tx=0x7fad74006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 msgr2=0x7fad80068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fad70005950 tx=0x7fad70005810 comp rx=0 tx=0).stop 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 shutdown_connections 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fad680387b0 0x7fad6803ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:18.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 --2- 172.21.15.64:0/3692085359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad800fe610 0x7fad80068400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:18.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 >> 172.21.15.64:0/3692085359 conn(0x7fad800f9ba0 msgr2=0x7fad800fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:18.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 shutdown_connections 2023-07-14T18:17:18.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:18.848+0000 7fad85216700 1 -- 172.21.15.64:0/3692085359 wait complete. 2023-07-14T18:17:19.246 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2023-07-14T18:17:19.247 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:17:19.247 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:17:19.267 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:17:19.267 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:19.326 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2023-07-14T18:17:19.327 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch host add smithi105 2023-07-14T18:17:19.894 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:20.154 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: Updating smithi064:/etc/ceph/ceph.conf 2023-07-14T18:17:20.155 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:19 smithi064 ceph-mon[96720]: Updating smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:21.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 -- 172.21.15.64:0/3246795174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 msgr2=0x7f68e01049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 --2- 172.21.15.64:0/3246795174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e01049e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f68d0009910 tx=0x7f68d0009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 -- 172.21.15.64:0/3246795174 shutdown_connections 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 --2- 172.21.15.64:0/3246795174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e01049e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 -- 172.21.15.64:0/3246795174 >> 172.21.15.64:0/3246795174 conn(0x7f68e0074310 msgr2=0x7f68e0074710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 -- 172.21.15.64:0/3246795174 shutdown_connections 2023-07-14T18:17:21.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.167+0000 7f68e511d700 1 -- 172.21.15.64:0/3246795174 wait complete. 2023-07-14T18:17:21.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.168+0000 7f68e511d700 1 Processor -- start 2023-07-14T18:17:21.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.168+0000 7f68e511d700 1 -- start start 2023-07-14T18:17:21.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68e511d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:21.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68e511d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e019ad50 con 0x7f68e0104610 2023-07-14T18:17:21.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68ded9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:21.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68ded9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45868/0 (socket says 172.21.15.64:45868) 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68ded9d700 1 -- 172.21.15.64:0/2614495071 learned_addr learned my addr 172.21.15.64:0/2614495071 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68ded9d700 1 -- 172.21.15.64:0/2614495071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00095f0 con 0x7f68e0104610 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68ded9d700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68d0004f40 tx=0x7f68d0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68d001d070 con 0x7f68e0104610 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f68d00044b0 con 0x7f68e0104610 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.169+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e019afb0 con 0x7f68e0104610 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.170+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f68d0003810 con 0x7f68e0104610 2023-07-14T18:17:21.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.170+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e0194890 con 0x7f68e0104610 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.170+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f68d0003a90 con 0x7f68e0104610 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.170+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e0061c70 con 0x7f68e0104610 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.171+0000 7f68c7fff700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 0x7f68c803ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.171+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f68d004c700 con 0x7f68e0104610 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.171+0000 7f68de59c700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 0x7f68c803ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:21.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.171+0000 7f68de59c700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 0x7f68c803ac40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68d4006fd0 tx=0x7f68d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:21.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.175+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f68d002a770 con 0x7f68e0104610 2023-07-14T18:17:21.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:20 smithi064 ceph-mon[96720]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:21.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:20 smithi064 ceph-mon[96720]: Updating smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:21.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:21.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:21.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:21.425 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:21.379+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) v1 -- 0x7f68e0103320 con 0x7f68c8038790 2023-07-14T18:17:22.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:21 smithi064 ceph-mon[96720]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:23.336 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:22 smithi064 ceph-mon[96720]: Deploying cephadm binary to smithi105 2023-07-14T18:17:24.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.229+0000 7f68c7fff700 1 -- 172.21.15.64:0/2614495071 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f68e0103320 con 0x7f68c8038790 2023-07-14T18:17:24.275 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi105' with addr '172.21.15.105' 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 msgr2=0x7f68c803ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 0x7f68c803ac40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f68d4006fd0 tx=0x7f68d4006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 msgr2=0x7f68e019a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f68d0004f40 tx=0x7f68d0005020 comp rx=0 tx=0).stop 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 shutdown_connections 2023-07-14T18:17:24.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f68c8038790 0x7f68c803ac40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:24.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 --2- 172.21.15.64:0/2614495071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f68e0104610 0x7f68e019a810 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:24.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.231+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 >> 172.21.15.64:0/2614495071 conn(0x7f68e0074310 msgr2=0x7f68e0068130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:24.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.232+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 shutdown_connections 2023-07-14T18:17:24.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:24.232+0000 7f68e511d700 1 -- 172.21.15.64:0/2614495071 wait complete. 2023-07-14T18:17:24.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:17:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:25.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:25 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:25.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:25 smithi064 ceph-mon[96720]: Added host smithi105 2023-07-14T18:17:25.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:25 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:25.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:25 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:26.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.356+0000 7f580d5ae700 1 -- 172.21.15.64:0/3592278452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 msgr2=0x7f58080ffc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.356+0000 7f580d5ae700 1 --2- 172.21.15.64:0/3592278452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58080ffc50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009910 tx=0x7f57f8009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.356+0000 7f580d5ae700 1 -- 172.21.15.64:0/3592278452 shutdown_connections 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.356+0000 7f580d5ae700 1 --2- 172.21.15.64:0/3592278452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58080ffc50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.356+0000 7f580d5ae700 1 -- 172.21.15.64:0/3592278452 >> 172.21.15.64:0/3592278452 conn(0x7f5808074e40 msgr2=0x7f5808075240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.357+0000 7f580d5ae700 1 -- 172.21.15.64:0/3592278452 shutdown_connections 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.357+0000 7f580d5ae700 1 -- 172.21.15.64:0/3592278452 wait complete. 2023-07-14T18:17:26.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.357+0000 7f580d5ae700 1 Processor -- start 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.357+0000 7f580d5ae700 1 -- start start 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f580d5ae700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f580d5ae700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5808195f10 con 0x7f58080ff880 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f5806ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f5806ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45894/0 (socket says 172.21.15.64:45894) 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f5806ffd700 1 -- 172.21.15.64:0/1575289626 learned_addr learned my addr 172.21.15.64:0/1575289626 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:26.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f5806ffd700 1 -- 172.21.15.64:0/1575289626 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80095f0 con 0x7f58080ff880 2023-07-14T18:17:26.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f5806ffd700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f57f8004f40 tx=0x7f57f8005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57f801d070 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f57f80044b0 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5808196110 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.358+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57f8003860 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.359+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58081964d0 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.359+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f57f800b960 con 0x7f58080ff880 2023-07-14T18:17:26.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.359+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5808106050 con 0x7f58080ff880 2023-07-14T18:17:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.360+0000 7f57effff700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 0x7f57f003acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.360+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57f804c640 con 0x7f58080ff880 2023-07-14T18:17:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.360+0000 7f58067fc700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 0x7f57f003acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:26.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.360+0000 7f58067fc700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 0x7f57f003acb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:26.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.364+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f57f802a540 con 0x7f58080ff880 2023-07-14T18:17:26.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:26.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:26 smithi064 ceph-mon[96720]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:26.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.565+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58080fed60 con 0x7f57f0038800 2023-07-14T18:17:26.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.566+0000 7f57effff700 1 -- 172.21.15.64:0/1575289626 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f58080fed60 con 0x7f57f0038800 2023-07-14T18:17:26.611 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:26.612 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2023-07-14T18:17:26.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 msgr2=0x7f57f003acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:26.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 0x7f57f003acb0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57fc006fd0 tx=0x7f57fc006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:26.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 msgr2=0x7f58081959d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:26.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f57f8004f40 tx=0x7f57f8005020 comp rx=0 tx=0).stop 2023-07-14T18:17:26.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 shutdown_connections 2023-07-14T18:17:26.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57f0038800 0x7f57f003acb0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:26.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 --2- 172.21.15.64:0/1575289626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58080ff880 0x7f58081959d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:26.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 >> 172.21.15.64:0/1575289626 conn(0x7f5808074e40 msgr2=0x7f5808103190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:26.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 shutdown_connections 2023-07-14T18:17:26.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:26.568+0000 7f580d5ae700 1 -- 172.21.15.64:0/1575289626 wait complete. 2023-07-14T18:17:26.980 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi144 2023-07-14T18:17:26.980 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:17:26.981 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:17:27.000 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:17:27.001 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:27.060 INFO:tasks.cephadm:Adding host smithi144 to orchestrator... 2023-07-14T18:17:27.061 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch host add smithi144 2023-07-14T18:17:27.616 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:27.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:27 smithi064 ceph-mon[96720]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:27.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:27 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:27.966 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:27 smithi064 ceph-mon[96720]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:28.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 -- 172.21.15.64:0/2214201831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 msgr2=0x7fdd24104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:28.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/2214201831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24104a30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009910 tx=0x7fdd20009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:28.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 -- 172.21.15.64:0/2214201831 shutdown_connections 2023-07-14T18:17:28.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/2214201831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24104a30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 -- 172.21.15.64:0/2214201831 >> 172.21.15.64:0/2214201831 conn(0x7fdd240743e0 msgr2=0x7fdd240747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 -- 172.21.15.64:0/2214201831 shutdown_connections 2023-07-14T18:17:28.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.642+0000 7fdd2b79b700 1 -- 172.21.15.64:0/2214201831 wait complete. 2023-07-14T18:17:28.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd2b79b700 1 Processor -- start 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd2b79b700 1 -- start start 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd2b79b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd2b79b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd24195d70 con 0x7fdd24104660 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd29537700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd29537700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45900/0 (socket says 172.21.15.64:45900) 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.643+0000 7fdd29537700 1 -- 172.21.15.64:0/1397521263 learned_addr learned my addr 172.21.15.64:0/1397521263 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:28.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd29537700 1 -- 172.21.15.64:0/1397521263 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200095f0 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd29537700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd20004f40 tx=0x7fdd20005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd2001d070 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd200044b0 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd24195fd0 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd20003810 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.644+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd241964d0 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.645+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd24103900 con 0x7fdd24104660 2023-07-14T18:17:28.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.645+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fdd2000b960 con 0x7fdd24104660 2023-07-14T18:17:28.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.645+0000 7fdd1a7fc700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 0x7fdd1003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:28.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.645+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd2004c630 con 0x7fdd24104660 2023-07-14T18:17:28.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.645+0000 7fdd28d36700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 0x7fdd1003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:28.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.646+0000 7fdd28d36700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 0x7fdd1003ac50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:28.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.650+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd2002a540 con 0x7fdd24104660 2023-07-14T18:17:28.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:28.851+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd24067f60 con 0x7fdd100387a0 2023-07-14T18:17:30.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:30.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: Deploying cephadm binary to smithi144 2023-07-14T18:17:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:30 smithi064 ceph-mon[96720]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:31.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:31 smithi064 ceph-mon[96720]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:31.586 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:31 smithi064 ceph-mon[96720]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:31.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.656+0000 7fdd1a7fc700 1 -- 172.21.15.64:0/1397521263 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fdd24067f60 con 0x7fdd100387a0 2023-07-14T18:17:31.702 INFO:teuthology.orchestra.run.smithi064.stdout:Added host 'smithi144' with addr '172.21.15.144' 2023-07-14T18:17:31.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 msgr2=0x7fdd1003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:31.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 0x7fdd1003ac50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdd14006fd0 tx=0x7fdd14006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 msgr2=0x7fdd24195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdd20004f40 tx=0x7fdd20005020 comp rx=0 tx=0).stop 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 shutdown_connections 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd100387a0 0x7fdd1003ac50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 --2- 172.21.15.64:0/1397521263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd24104660 0x7fdd24195830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 >> 172.21.15.64:0/1397521263 conn(0x7fdd240743e0 msgr2=0x7fdd24105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 shutdown_connections 2023-07-14T18:17:31.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:31.659+0000 7fdd2b79b700 1 -- 172.21.15.64:0/1397521263 wait complete. 2023-07-14T18:17:32.069 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:17:32.630 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:32.817 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:32.817 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:32.817 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:32.817 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:32.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:32.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:32.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:32.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:33.681 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:33 smithi064 ceph-mon[96720]: Added host smithi144 2023-07-14T18:17:33.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:33.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.771+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/2988285326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 msgr2=0x7fdd64105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.771+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/2988285326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd64105af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdd5400b0a0 tx=0x7fdd5400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.771+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/2988285326 shutdown_connections 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.771+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/2988285326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd64105af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.772+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/2988285326 >> 172.21.15.64:0/2988285326 conn(0x7fdd640fb260 msgr2=0x7fdd640fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.772+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/2988285326 shutdown_connections 2023-07-14T18:17:33.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.772+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/2988285326 wait complete. 2023-07-14T18:17:33.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.772+0000 7fdd6b1cc700 1 Processor -- start 2023-07-14T18:17:33.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.772+0000 7fdd6b1cc700 1 -- start start 2023-07-14T18:17:33.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd6b1cc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:33.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd6b1cc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6419ad40 con 0x7fdd64105720 2023-07-14T18:17:33.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd68f68700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd68f68700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45926/0 (socket says 172.21.15.64:45926) 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd68f68700 1 -- 172.21.15.64:0/491430322 learned_addr learned my addr 172.21.15.64:0/491430322 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd68f68700 1 -- 172.21.15.64:0/491430322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd54009d00 con 0x7fdd64105720 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd68f68700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009fd0 tx=0x7fdd5400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd54021070 con 0x7fdd64105720 2023-07-14T18:17:33.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdd54003d40 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6419af40 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.773+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd5400e040 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.774+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd64194880 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.774+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fdd54004040 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.774+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd64061c70 con 0x7fdd64105720 2023-07-14T18:17:33.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.775+0000 7fdd61ffb700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 0x7fdd4c03acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:33.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.775+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd5404c440 con 0x7fdd64105720 2023-07-14T18:17:33.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.775+0000 7fdd63fff700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 0x7fdd4c03acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:33.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.775+0000 7fdd63fff700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 0x7fdd4c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdd58006fd0 tx=0x7fdd58006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:33.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.780+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdd54025790 con 0x7fdd64105720 2023-07-14T18:17:34.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.982+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd641952b0 con 0x7fdd4c038800 2023-07-14T18:17:34.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.983+0000 7fdd61ffb700 1 -- 172.21.15.64:0/491430322 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fdd641952b0 con 0x7fdd4c038800 2023-07-14T18:17:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:17:34.029 INFO:teuthology.orchestra.run.smithi064.stdout:[{"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.144", "hostname": "smithi144", "labels": [], "status": ""}] 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 msgr2=0x7fdd4c03acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 0x7fdd4c03acb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fdd58006fd0 tx=0x7fdd58006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 msgr2=0x7fdd6419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdd54009fd0 tx=0x7fdd5400bd40 comp rx=0 tx=0).stop 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 shutdown_connections 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdd4c038800 0x7fdd4c03acb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:34.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 --2- 172.21.15.64:0/491430322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd64105720 0x7fdd6419a800 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:34.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 >> 172.21.15.64:0/491430322 conn(0x7fdd640fb260 msgr2=0x7fdd640fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:34.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 shutdown_connections 2023-07-14T18:17:34.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:33.985+0000 7fdd6b1cc700 1 -- 172.21.15.64:0/491430322 wait complete. 2023-07-14T18:17:34.375 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T18:17:34.376 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T18:17:34.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:34 smithi064 ceph-mon[96720]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:34.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:34 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:34.934 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:35.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:35 smithi064 ceph-mon[96720]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 -- 172.21.15.64:0/2801425488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 msgr2=0x7fcacc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 --2- 172.21.15.64:0/2801425488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc104a30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcac0009910 tx=0x7fcac0009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 -- 172.21.15.64:0/2801425488 shutdown_connections 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 --2- 172.21.15.64:0/2801425488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc104a30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 -- 172.21.15.64:0/2801425488 >> 172.21.15.64:0/2801425488 conn(0x7fcacc0fa140 msgr2=0x7fcacc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:36.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 -- 172.21.15.64:0/2801425488 shutdown_connections 2023-07-14T18:17:36.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.115+0000 7fcad473f700 1 -- 172.21.15.64:0/2801425488 wait complete. 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad473f700 1 Processor -- start 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad473f700 1 -- start start 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad473f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad473f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcacc195da0 con 0x7fcacc104660 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad24db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:36.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad24db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45932/0 (socket says 172.21.15.64:45932) 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad24db700 1 -- 172.21.15.64:0/1905126558 learned_addr learned my addr 172.21.15.64:0/1905126558 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.116+0000 7fcad24db700 1 -- 172.21.15.64:0/1905126558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac00095f0 con 0x7fcacc104660 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcad24db700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006010 tx=0x7fcac0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcac001d070 con 0x7fcacc104660 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcac000bdf0 con 0x7fcacc104660 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcacc196000 con 0x7fcacc104660 2023-07-14T18:17:36.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcac0022d80 con 0x7fcacc104660 2023-07-14T18:17:36.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.117+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcacc196500 con 0x7fcacc104660 2023-07-14T18:17:36.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fcac000f460 con 0x7fcacc104660 2023-07-14T18:17:36.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcabf7fe700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 0x7fcab803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:36.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcac004c840 con 0x7fcacc104660 2023-07-14T18:17:36.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcacc103900 con 0x7fcacc104660 2023-07-14T18:17:36.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcad1cda700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 0x7fcab803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:36.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.118+0000 7fcad1cda700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 0x7fcab803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcac8006fd0 tx=0x7fcac8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:36.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.123+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcac0027030 con 0x7fcacc104660 2023-07-14T18:17:36.369 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.324+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcacc0686c0 con 0x7fcacc104660 2023-07-14T18:17:36.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.397+0000 7fcabf7fe700 1 -- 172.21.15.64:0/1905126558 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcac0005c00 con 0x7fcacc104660 2023-07-14T18:17:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 msgr2=0x7fcab803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 0x7fcab803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcac8006fd0 tx=0x7fcac8006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 msgr2=0x7fcacc195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcac0006010 tx=0x7fcac0004ff0 comp rx=0 tx=0).stop 2023-07-14T18:17:36.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 shutdown_connections 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcab8038800 0x7fcab803acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 --2- 172.21.15.64:0/1905126558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcacc104660 0x7fcacc195860 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 >> 172.21.15.64:0/1905126558 conn(0x7fcacc0fa140 msgr2=0x7fcacc0fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 shutdown_connections 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:36.399+0000 7fcad473f700 1 -- 172.21.15.64:0/1905126558 wait complete. 2023-07-14T18:17:36.446 INFO:teuthology.orchestra.run.smithi064.stderr:adjusted tunables profile to default 2023-07-14T18:17:36.798 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:36 smithi064 ceph-mon[96720]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:36.799 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:36 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:17:36.898 INFO:tasks.cephadm:Adding mon.a on smithi064 2023-07-14T18:17:36.898 INFO:tasks.cephadm:Adding mon.b on smithi105 2023-07-14T18:17:36.898 INFO:tasks.cephadm:Adding mon.c on smithi144 2023-07-14T18:17:36.899 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c' 2023-07-14T18:17:37.483 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:17:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:37.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:38.740 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- 172.21.15.144:0/1638224835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 msgr2=0x7f6c20104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 --2- 172.21.15.144:0/1638224835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c20104a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009910 tx=0x7f6c14009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- 172.21.15.144:0/1638224835 shutdown_connections 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 --2- 172.21.15.144:0/1638224835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c20104a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- 172.21.15.144:0/1638224835 >> 172.21.15.144:0/1638224835 conn(0x7f6c200743e0 msgr2=0x7f6c200747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- 172.21.15.144:0/1638224835 shutdown_connections 2023-07-14T18:17:38.741 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- 172.21.15.144:0/1638224835 wait complete. 2023-07-14T18:17:38.742 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 Processor -- start 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.695+0000 7f6c26b93700 1 -- start start 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c26b93700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c26b93700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2019ad60 con 0x7f6c20104660 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c2492f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c2492f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.144:42028/0 (socket says 172.21.15.144:42028) 2023-07-14T18:17:38.743 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c2492f700 1 -- 172.21.15.144:0/4174708899 learned_addr learned my addr 172.21.15.144:0/4174708899 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c2492f700 1 -- 172.21.15.144:0/4174708899 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c140095f0 con 0x7f6c20104660 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c2492f700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c14004f40 tx=0x7f6c14005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c1401d070 con 0x7f6c20104660 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c140044b0 con 0x7f6c20104660 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2019afc0 con 0x7f6c20104660 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.696+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6c14003810 con 0x7f6c20104660 2023-07-14T18:17:38.744 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.697+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c201948a0 con 0x7f6c20104660 2023-07-14T18:17:38.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.697+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6c1400b960 con 0x7f6c20104660 2023-07-14T18:17:38.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.697+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c20061c70 con 0x7f6c20104660 2023-07-14T18:17:38.745 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.698+0000 7f6c1dffb700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 0x7f6c1003ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:38.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.698+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6c1404c670 con 0x7f6c20104660 2023-07-14T18:17:38.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.698+0000 7f6c1ffff700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 0x7f6c1003ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:38.746 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.698+0000 7f6c1ffff700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 0x7f6c1003ac50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c006fd0 tx=0x7f6c0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:38.748 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.703+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c1402a930 con 0x7f6c20104660 2023-07-14T18:17:38.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:38.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:38.942 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.896+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c20103430 con 0x7f6c100387a0 2023-07-14T18:17:38.952 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.906+0000 7f6c1dffb700 1 -- 172.21.15.144:0/4174708899 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6c20103430 con 0x7f6c100387a0 2023-07-14T18:17:38.952 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mon update... 2023-07-14T18:17:38.954 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 msgr2=0x7f6c1003ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:38.954 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 0x7f6c1003ac50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c006fd0 tx=0x7f6c0c006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:38.954 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 msgr2=0x7f6c2019a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:38.954 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6c14004f40 tx=0x7f6c14005020 comp rx=0 tx=0).stop 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 shutdown_connections 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6c100387a0 0x7f6c1003ac50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 --2- 172.21.15.144:0/4174708899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20104660 0x7f6c2019a820 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.908+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 >> 172.21.15.144:0/4174708899 conn(0x7f6c200743e0 msgr2=0x7f6c200680d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.909+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 shutdown_connections 2023-07-14T18:17:38.955 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:38.909+0000 7f6c26b93700 1 -- 172.21.15.144:0/4174708899 wait complete. 2023-07-14T18:17:39.352 DEBUG:teuthology.orchestra.run.smithi105:mon.b> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T18:17:39.354 DEBUG:teuthology.orchestra.run.smithi144:mon.c> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T18:17:39.360 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:17:39.360 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:17:39.388 INFO:journalctl@ceph.mon.c.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:17:39.390 INFO:journalctl@ceph.mon.b.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:17:39.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:39.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:40.312 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:40.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:40 smithi064 ceph-mon[96720]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:40.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:40 smithi064 ceph-mon[96720]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c;count:3 2023-07-14T18:17:40.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:40 smithi064 ceph-mon[96720]: Deploying daemon mon.c on smithi144 2023-07-14T18:17:40.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:40 smithi064 ceph-mon[96720]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:42.203 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 -- 172.21.15.144:0/1711146982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 msgr2=0x7fe0dc0feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:42.203 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 --2- 172.21.15.144:0/1711146982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc0feb40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009910 tx=0x7fe0d0009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:42.203 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 -- 172.21.15.144:0/1711146982 shutdown_connections 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 --2- 172.21.15.144:0/1711146982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc0feb40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 -- 172.21.15.144:0/1711146982 >> 172.21.15.144:0/1711146982 conn(0x7fe0dc074e40 msgr2=0x7fe0dc075240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 -- 172.21.15.144:0/1711146982 shutdown_connections 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.156+0000 7fe0e23db700 1 -- 172.21.15.144:0/1711146982 wait complete. 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.157+0000 7fe0e23db700 1 Processor -- start 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.157+0000 7fe0e23db700 1 -- start start 2023-07-14T18:17:42.204 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.157+0000 7fe0e23db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:42.205 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.157+0000 7fe0e23db700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0dc19aed0 con 0x7fe0dc0fe770 2023-07-14T18:17:42.205 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0dbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:42.205 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0dbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.144:42034/0 (socket says 172.21.15.144:42034) 2023-07-14T18:17:42.205 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0dbfff700 1 -- 172.21.15.144:0/2937610462 learned_addr learned my addr 172.21.15.144:0/2937610462 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:17:42.206 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0dbfff700 1 -- 172.21.15.144:0/2937610462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d00095f0 con 0x7fe0dc0fe770 2023-07-14T18:17:42.206 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0dbfff700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0004f40 tx=0x7fe0d0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:42.206 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0d001d070 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0dc19b0d0 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe0d00044b0 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe0d0003860 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.158+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0dc19b490 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.159+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0dc061c70 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.159+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe0d000b960 con 0x7fe0dc0fe770 2023-07-14T18:17:42.207 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.159+0000 7fe0d97fa700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 0x7fe0c403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:42.208 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.160+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe0d004c630 con 0x7fe0dc0fe770 2023-07-14T18:17:42.208 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.160+0000 7fe0db7fe700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 0x7fe0c403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:42.208 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.160+0000 7fe0db7fe700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 0x7fe0c403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc006fd0 tx=0x7fe0cc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:42.210 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.165+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe0d002a540 con 0x7fe0dc0fe770 2023-07-14T18:17:42.509 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.462+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0dc195f20 con 0x7fe0dc0fe770 2023-07-14T18:17:42.510 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.463+0000 7fe0d97fa700 1 -- 172.21.15.144:0/2937610462 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7fe0d002a360 con 0x7fe0dc0fe770 2023-07-14T18:17:42.510 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:17:42.510 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:16:01.841108Z","created":"2023-07-14T18:16:01.841108Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T18:17:42.511 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.465+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 msgr2=0x7fe0c403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:42.511 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.465+0000 7fe0e23db700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 0x7fe0c403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc006fd0 tx=0x7fe0cc006e40 comp rx=0 tx=0).stop 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.465+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 msgr2=0x7fe0dc19a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.465+0000 7fe0e23db700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0004f40 tx=0x7fe0d0005020 comp rx=0 tx=0).stop 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 shutdown_connections 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0c40387b0 0x7fe0c403ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 --2- 172.21.15.144:0/2937610462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0dc0fe770 0x7fe0dc19a990 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:42.512 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 >> 172.21.15.144:0/2937610462 conn(0x7fe0dc074e40 msgr2=0x7fe0dc104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:42.513 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 shutdown_connections 2023-07-14T18:17:42.513 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:42.466+0000 7fe0e23db700 1 -- 172.21.15.144:0/2937610462 wait complete. 2023-07-14T18:17:42.513 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2023-07-14T18:17:42.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:42 smithi064 ceph-mon[96720]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.177 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 podman[98375]: 2023-07-14T18:17:43.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pidfile_write: ignore empty --pid-file 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: load: jerasure load: lrc 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Git sha 0 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:17:43.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: DB SUMMARY 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: DB Session ID: 17AJ7DE3VJH5APWWVC9X 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:17:43.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.env: 0x558424aa8c60 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:17:43.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.info_log: 0x558426a6d2a0 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.statistics: (nil) 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.use_fsync: 0 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:17:43.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:17:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.db_log_dir: 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.wal_dir: 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:17:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.write_buffer_manager: 0x55842698e3c0 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:17:43.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.unordered_write: 0 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:17:43.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.row_cache: None 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.wal_filter: None 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:17:43.686 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.wal_compression: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:17:43.687 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:17:43.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:17:43.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:17:43.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:17:43.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:17:43.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:17:43.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:17:43.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_open_files: -1 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:17:43.691 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Compression algorithms supported: 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kZSTD supported: 0 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:17:43.692 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:17:43.693 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.merge_operator: 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_filter: None 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558426a6d3a0) 2023-07-14T18:17:43.694 INFO:journalctl@ceph.mon.c.smithi144.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: index_type: 0 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: data_block_index_type: 0 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: index_shortening: 1 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:17:43.695 INFO:journalctl@ceph.mon.c.smithi144.stdout: checksum: 4 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: no_block_cache: 0 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_cache: 0x558426a2d1f0 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_cache_options: 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: capacity : 536870912 2023-07-14T18:17:43.696 INFO:journalctl@ceph.mon.c.smithi144.stdout: num_shard_bits : 4 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: strict_capacity_limit : 0 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_cache_compressed: (nil) 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: persistent_cache: (nil) 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_size: 4096 2023-07-14T18:17:43.697 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_size_deviation: 10 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_restart_interval: 16 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: index_block_restart_interval: 1 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: metadata_block_size: 4096 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: partition_filters: 0 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: use_delta_encoding: 1 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: filter_policy: bloomfilter 2023-07-14T18:17:43.698 INFO:journalctl@ceph.mon.c.smithi144.stdout: whole_key_filtering: 1 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: verify_compression: 0 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: format_version: 5 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: enable_index_compression: 1 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: block_align: 0 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: max_auto_readahead_size: 262144 2023-07-14T18:17:43.699 INFO:journalctl@ceph.mon.c.smithi144.stdout: prepopulate_block_cache: 0 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 bash[98375]: dc66a4d8a24eb9f4951c99df1bbc452452939e4bfca6ea0a1435e3319a341eed 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression: NoCompression 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:17:43.700 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.num_levels: 7 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:17:43.701 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:17:43.702 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:17:43.703 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:17:43.704 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:17:43.704 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:17:43.704 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:17:43.705 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:17:43.705 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:17:43.705 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:17:43.705 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:17:43.705 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:17:43.706 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:17:43.707 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:17:43.708 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:17:43.709 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:17:43.710 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.ttl: 2592000 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:17:43.711 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T18:17:43.712 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 7694cfd1-6160-4467-bfba-278a1d85f6e4 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358663246942, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358663247344, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689358663, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "7694cfd1-6160-4467-bfba-278a1d85f6e4", "db_session_id": "17AJ7DE3VJH5APWWVC9X", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358663247419, "job": 1, "event": "recovery_finished"} 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T18:17:43.713 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558426ac6000 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: DB pointer 0x558426ab2000 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout: ** DB Stats ** 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:43.714 INFO:journalctl@ceph.mon.c.smithi144.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: ** Compaction Stats [default] ** 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:17:43.715 INFO:journalctl@ceph.mon.c.smithi144.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: ** Compaction Stats [default] ** 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:17:43.716 INFO:journalctl@ceph.mon.c.smithi144.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:17:43.717 INFO:journalctl@ceph.mon.c.smithi144.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: Interval compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: Block cache BinnedLRUCache@0x558426a2d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T18:17:43.718 INFO:journalctl@ceph.mon.c.smithi144.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: using public_addr v2:172.21.15.144:0/0 -> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: starting mon.c rank -1 at public addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] at bind addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(???) e0 preinit fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T18:17:43.719 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: e1 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: legacy client fscid: -1 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: 2023-07-14T18:17:43.720 INFO:journalctl@ceph.mon.c.smithi144.stdout: No filesystems configured 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:17:43.721 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.722 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Deploying cephadm binary to smithi144 2023-07-14T18:17:43.723 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.724 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:43.724 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:43.724 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:43.724 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.725 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Added host smithi144 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:17:43.726 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:43.727 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.728 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:43.729 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c;count:3 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: Deploying daemon mon.c on smithi144 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T18:17:43.730 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expand map: {default=false} 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expand map: {default=info} 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:17:43.731 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:17:43.732 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:17:43.732 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:17:43.732 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:17:43.732 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 ceph-mon[98417]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:17:43.732 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:43 smithi144 systemd[1]: Started Ceph mon.c for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:17:43.940 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:17:43.940 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:17:44.492 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:17:48.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: mon.a calling monitor election 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: mon.c calling monitor election 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:48.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: fsmap 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: mgrmap e13: x(active, since 42s) 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: overall HEALTH_OK 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:48.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 -- 172.21.15.144:0/2270077962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94680035c0 msgr2=0x7f9468005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 --2- 172.21.15.144:0/2270077962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94680035c0 0x7f9468005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f947c067dc0 tx=0x7f946c00feb0 comp rx=0 tx=0).stop 2023-07-14T18:17:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 -- 172.21.15.144:0/2270077962 shutdown_connections 2023-07-14T18:17:48.818 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 --2- 172.21.15.144:0/2270077962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94680035c0 0x7f9468005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:48.819 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 -- 172.21.15.144:0/2270077962 >> 172.21.15.144:0/2270077962 conn(0x7f947c074500 msgr2=0x7f947c074900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:48.819 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 -- 172.21.15.144:0/2270077962 shutdown_connections 2023-07-14T18:17:48.819 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.770+0000 7f94821c3700 1 -- 172.21.15.144:0/2270077962 wait complete. 2023-07-14T18:17:48.820 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 Processor -- start 2023-07-14T18:17:48.820 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 -- start start 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 0x7f947c194b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c19b530 con 0x7f947c19b000 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.772+0000 7f94821c3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f947c19b6a0 con 0x7f94680035c0 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54672/0 (socket says 172.21.15.144:54672) 2023-07-14T18:17:48.821 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 -- 172.21.15.144:0/1327227429 learned_addr learned my addr 172.21.15.144:0/1327227429 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947affd700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 0x7f947c194b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 -- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 msgr2=0x7f947c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 -- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 msgr2=0x7f947c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947b7fe700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947affd700 1 -- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 msgr2=0x7f947c19aac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:17:48.822 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947affd700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947affd700 1 -- 172.21.15.144:0/1327227429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c00f900 con 0x7f947c19b000 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f947affd700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 0x7f947c194b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f946400b500 tx=0x7f946400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.773+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f9464011630 con 0x7f947c19b000 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.774+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9464011c20 con 0x7f947c19b000 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.774+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f947c195140 con 0x7f947c19b000 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.774+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f9464010c40 con 0x7f947c19b000 2023-07-14T18:17:48.823 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.774+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f947c195690 con 0x7f947c19b000 2023-07-14T18:17:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.775+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f946401a410 con 0x7f947c19b000 2023-07-14T18:17:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.775+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c1959c0 con 0x7f947c19b000 2023-07-14T18:17:48.824 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.775+0000 7f9478ff9700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 0x7f945c03aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:48.825 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.775+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f946404a820 con 0x7f947c19b000 2023-07-14T18:17:48.825 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.775+0000 7f947b7fe700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 0x7f945c03aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:48.825 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.776+0000 7f947b7fe700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 0x7f945c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f946c005b40 tx=0x7f946c005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:48.827 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:48.780+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9464010da0 con 0x7f947c19b000 2023-07-14T18:17:48.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: mon.a calling monitor election 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: mon.c calling monitor election 2023-07-14T18:17:48.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: fsmap 2023-07-14T18:17:48.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: mgrmap e13: x(active, since 42s) 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: overall HEALTH_OK 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:49.091 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.045+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f947c061c70 con 0x7f947c19b000 2023-07-14T18:17:49.092 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.045+0000 7f9478ff9700 1 -- 172.21.15.144:0/1327227429 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f9464018310 con 0x7f947c19b000 2023-07-14T18:17:49.092 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:17:49.093 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":2,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:17:43.274590Z","created":"2023-07-14T18:16:01.841108Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T18:17:49.094 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 msgr2=0x7f945c03aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:49.094 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 0x7f945c03aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f946c005b40 tx=0x7f946c005a90 comp rx=0 tx=0).stop 2023-07-14T18:17:49.094 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 msgr2=0x7f947c194b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 0x7f947c194b40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f946400b500 tx=0x7f946400b8c0 comp rx=0 tx=0).stop 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 shutdown_connections 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f94680035c0 0x7f947c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f945c0385f0 0x7f945c03aaa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 --2- 172.21.15.144:0/1327227429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f947c19b000 0x7f947c194b40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 >> 172.21.15.144:0/1327227429 conn(0x7f947c074500 msgr2=0x7f947c101960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:49.095 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 shutdown_connections 2023-07-14T18:17:49.096 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:49.047+0000 7f94821c3700 1 -- 172.21.15.144:0/1327227429 wait complete. 2023-07-14T18:17:49.096 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 2 2023-07-14T18:17:49.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:49 smithi064 ceph-mon[96720]: Deploying daemon mon.b on smithi105 2023-07-14T18:17:49.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:49 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/1327227429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:49.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:49.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:49 smithi144 ceph-mon[98417]: Deploying daemon mon.b on smithi105 2023-07-14T18:17:49.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:49 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/1327227429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:49.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:50.543 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:17:50.543 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:17:50.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:50 smithi064 ceph-mon[96720]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:50.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:50 smithi144 ceph-mon[98417]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.105 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:17:51.189 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:50 smithi105 systemd[1]: Starting Ceph mon.b for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:17:51.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 podman[97973]: 2023-07-14T18:17:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:17:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:17:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pidfile_write: ignore empty --pid-file 2023-07-14T18:17:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: load: jerasure load: lrc 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Git sha 0 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: DB SUMMARY 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: DB Session ID: PKICJV6S6IP3Y7B20QOR 2023-07-14T18:17:51.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:17:51.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.env: 0x562f26a47c60 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:17:51.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.info_log: 0x562f27a6f2a0 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.statistics: (nil) 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.use_fsync: 0 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:17:51.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.db_log_dir: 2023-07-14T18:17:51.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.wal_dir: 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:17:51.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.write_buffer_manager: 0x562f279903c0 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:17:51.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.unordered_write: 0 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:17:51.565 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.row_cache: None 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.wal_filter: None 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.wal_compression: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:17:51.566 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:17:51.567 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:17:51.568 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:17:51.569 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:17:51.569 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:17:51.569 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:17:51.569 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:17:51.569 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:17:51.570 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_open_files: -1 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Compression algorithms supported: 2023-07-14T18:17:51.571 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kZSTD supported: 0 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:17:51.572 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.merge_operator: 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_filter: None 2023-07-14T18:17:51.573 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562f27a6f3a0) 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:17:51.574 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_type: 0 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_index_type: 0 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_shortening: 1 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: checksum: 4 2023-07-14T18:17:51.575 INFO:journalctl@ceph.mon.b.smithi105.stdout: no_block_cache: 0 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache: 0x562f27a2f1f0 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_options: 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: capacity : 536870912 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_shard_bits : 4 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: strict_capacity_limit : 0 2023-07-14T18:17:51.576 INFO:journalctl@ceph.mon.b.smithi105.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_compressed: (nil) 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: persistent_cache: (nil) 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size: 4096 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size_deviation: 10 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_restart_interval: 16 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_block_restart_interval: 1 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: metadata_block_size: 4096 2023-07-14T18:17:51.577 INFO:journalctl@ceph.mon.b.smithi105.stdout: partition_filters: 0 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: use_delta_encoding: 1 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: filter_policy: bloomfilter 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: whole_key_filtering: 1 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: verify_compression: 0 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: format_version: 5 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_index_compression: 1 2023-07-14T18:17:51.578 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_align: 0 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout: max_auto_readahead_size: 262144 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout: prepopulate_block_cache: 0 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression: NoCompression 2023-07-14T18:17:51.579 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.num_levels: 7 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:17:51.580 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:17:51.581 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 bash[97973]: 4f07c1d47ce2435e124ab869512fadf60b0b15e8f2c44c4c5fb67659740eb63a 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:17:51.582 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:17:51.583 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:17:51.583 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:17:51.583 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:17:51.583 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:17:51.583 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:17:51.584 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:17:51.585 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:17:51.586 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:17:51.587 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:17:51.588 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.ttl: 2592000 2023-07-14T18:17:51.589 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:17:51.590 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T18:17:51.591 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9e5c5cc3-d64a-43c1-9954-d944ddcfcfc2 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358671355062, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358671355473, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689358671, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9e5c5cc3-d64a-43c1-9954-d944ddcfcfc2", "db_session_id": "PKICJV6S6IP3Y7B20QOR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689358671355540, "job": 1, "event": "recovery_finished"} 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562f27ac8000 2023-07-14T18:17:51.592 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: DB pointer 0x562f27ab4000 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** DB Stats ** 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:17:51.593 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:17:51.594 INFO:journalctl@ceph.mon.b.smithi105.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:17:51.595 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:17:51.596 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache BinnedLRUCache@0x562f27a2f1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:17:51.597 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: using public_addr v2:172.21.15.105:0/0 -> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: starting mon.b rank -1 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(???) e0 preinit fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 systemd[1]: Started Ceph mon.b for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:17:51.598 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: e1 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: legacy client fscid: -1 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout: No filesystems configured 2023-07-14T18:17:51.599 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:17:51.600 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Deploying cephadm binary to smithi144 2023-07-14T18:17:51.602 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.603 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Added host smithi144 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.604 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1905126558' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.605 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:17:51.606 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:51.607 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Saving service mon spec with placement smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c;count:3 2023-07-14T18:17:51.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Deploying daemon mon.c on smithi144 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.a calling monitor election 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.609 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.c calling monitor election 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:51.610 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: monmap e2: 2 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: fsmap 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mgrmap e13: x(active, since 42s) 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: overall HEALTH_OK 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:51.611 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: Deploying daemon mon.b on smithi105 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/1327227429' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T18:17:51.612 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expand map: {default=false} 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expand map: {default=info} 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:17:51.613 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:17:51.614 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:17:51.614 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:17:51.614 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:17:51.614 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:51 smithi105 ceph-mon[98013]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:17:56.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: mon.c calling monitor election 2023-07-14T18:17:56.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:56.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: mon.a calling monitor election 2023-07-14T18:17:56.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: fsmap 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: mgrmap e13: x(active, since 51s) 2023-07-14T18:17:56.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.813+0000 7f8b190c1700 1 -- 172.21.15.144:0/3554846656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8afc0057d0 msgr2=0x7f8afc007c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.813+0000 7f8b190c1700 1 --2- 172.21.15.144:0/3554846656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8afc0057d0 0x7f8afc007c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8b14077a10 tx=0x7f8b0800fe60 comp rx=0 tx=0).stop 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.813+0000 7f8b190c1700 1 -- 172.21.15.144:0/3554846656 shutdown_connections 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.813+0000 7f8b190c1700 1 --2- 172.21.15.144:0/3554846656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8afc0057d0 0x7f8afc007c50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.813+0000 7f8b190c1700 1 -- 172.21.15.144:0/3554846656 >> 172.21.15.144:0/3554846656 conn(0x7f8b14073ea0 msgr2=0x7f8b140742a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:56.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.814+0000 7f8b190c1700 1 -- 172.21.15.144:0/3554846656 shutdown_connections 2023-07-14T18:17:56.862 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.814+0000 7f8b190c1700 1 -- 172.21.15.144:0/3554846656 wait complete. 2023-07-14T18:17:56.863 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 Processor -- start 2023-07-14T18:17:56.863 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 -- start start 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 0x7f8b14193440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b14199530 con 0x7f8b14192fa0 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1419f800 con 0x7f8b14198f20 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b190c1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1419f940 con 0x7f8afc0057d0 2023-07-14T18:17:56.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54698/0 (socket says 172.21.15.144:54698) 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 -- 172.21.15.144:0/1227456973 learned_addr learned my addr 172.21.15.144:0/1227456973 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 msgr2=0x7f8b141989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 msgr2=0x7f8b141989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b1359e700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 0x7f8b14193440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:56.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b12d9d700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.815+0000 7f8b1259c700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1259c700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 msgr2=0x7f8b14192a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1259c700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 msgr2=0x7f8b14192a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1259c700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1259c700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:17:56.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 msgr2=0x7f8b141989e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 msgr2=0x7f8b14192a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 -- 172.21.15.144:0/1227456973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0800f8b0 con 0x7f8b14192fa0 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b1359e700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 0x7f8b14193440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00bd50 tx=0x7f8b0c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00cd10 con 0x7f8b14192fa0 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b0c00ce70 con 0x7f8b14192fa0 2023-07-14T18:17:56.867 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1419fc20 con 0x7f8b14192fa0 2023-07-14T18:17:56.868 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c012710 con 0x7f8b14192fa0 2023-07-14T18:17:56.868 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.816+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b141a0120 con 0x7f8b14192fa0 2023-07-14T18:17:56.868 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.817+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f8b0c014440 con 0x7f8b14192fa0 2023-07-14T18:17:56.869 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.817+0000 7f8afbfff700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 0x7f8b0003b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:56.869 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.817+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8b0c04adb0 con 0x7f8b14192fa0 2023-07-14T18:17:56.869 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.817+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b1418d040 con 0x7f8b14192fa0 2023-07-14T18:17:56.869 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.818+0000 7f8b12d9d700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 0x7f8b0003b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:56.869 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.818+0000 7f8b12d9d700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 0x7f8b0003b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b08005b40 tx=0x7f8b08005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:56.870 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:56.823+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b0c0146f0 con 0x7f8b14192fa0 2023-07-14T18:17:56.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: mon.c calling monitor election 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: mon.a calling monitor election 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:56.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: fsmap 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: mgrmap e13: x(active, since 51s) 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2023-07-14T18:17:56.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:57.139 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.091+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b1418ca20 con 0x7f8b14192fa0 2023-07-14T18:17:57.139 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.092+0000 7f8afbfff700 1 -- 172.21.15.144:0/1227456973 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7f8b0c01b6e0 con 0x7f8b14192fa0 2023-07-14T18:17:57.140 INFO:teuthology.orchestra.run.smithi144.stdout: 2023-07-14T18:17:57.140 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":3,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:17:51.387080Z","created":"2023-07-14T18:16:01.841108Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 msgr2=0x7f8b0003b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 0x7f8b0003b0e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8b08005b40 tx=0x7f8b08005a90 comp rx=0 tx=0).stop 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 msgr2=0x7f8b14193440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 0x7f8b14193440 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c00bd50 tx=0x7f8b0c00be30 comp rx=0 tx=0).stop 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 shutdown_connections 2023-07-14T18:17:57.142 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8afc0057d0 0x7f8b141989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8b00038c30 0x7f8b0003b0e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b14192fa0 0x7f8b14193440 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 --2- 172.21.15.144:0/1227456973 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8b14198f20 0x7f8b14192a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 >> 172.21.15.144:0/1227456973 conn(0x7f8b14073ea0 msgr2=0x7f8b14100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 shutdown_connections 2023-07-14T18:17:57.143 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:17:57.095+0000 7f8b190c1700 1 -- 172.21.15.144:0/1227456973 wait complete. 2023-07-14T18:17:57.144 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 3 2023-07-14T18:17:57.572 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T18:17:57.573 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T18:17:57.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:57 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:57.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:57 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/1227456973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:57.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:57 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:57.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:57 smithi064 ceph-mon[96720]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:57.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:57 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:57.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:57 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/1227456973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:57.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:57 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:57.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:57 smithi144 ceph-mon[98417]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:58.147 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:17:58.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: mon.c calling monitor election 2023-07-14T18:17:58.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:17:58.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: mon.a calling monitor election 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:58.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: fsmap 2023-07-14T18:17:58.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: mgrmap e13: x(active, since 51s) 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:58.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/1227456973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:17:58.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:58 smithi105 ceph-mon[98013]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:17:59.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 -- 172.21.15.64:0/126460832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b81057c0 msgr2=0x7f13b8105b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:59.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 --2- 172.21.15.64:0/126460832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b81057c0 0x7f13b8105b90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009910 tx=0x7f13a8009c20 comp rx=0 tx=0).stop 2023-07-14T18:17:59.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 -- 172.21.15.64:0/126460832 shutdown_connections 2023-07-14T18:17:59.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 --2- 172.21.15.64:0/126460832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b81057c0 0x7f13b8105b90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 -- 172.21.15.64:0/126460832 >> 172.21.15.64:0/126460832 conn(0x7f13b80743e0 msgr2=0x7f13b80747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 -- 172.21.15.64:0/126460832 shutdown_connections 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.272+0000 7f13bcad1700 1 -- 172.21.15.64:0/126460832 wait complete. 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 Processor -- start 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 -- start start 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b81057c0 0x7f13b819aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:59.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 0x7f13b8194b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b819b610 con 0x7f13b8195080 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b81a18e0 con 0x7f13b819b000 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13bcad1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b81a1a20 con 0x7f13b81057c0 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13b6d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13b6d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45938/0 (socket says 172.21.15.64:45938) 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13b6d9d700 1 -- 172.21.15.64:0/2522820924 learned_addr learned my addr 172.21.15.64:0/2522820924 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:17:59.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13b659c700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b81057c0 0x7f13b819aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.273+0000 7f13b5d9b700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 0x7f13b8194b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 -- 172.21.15.64:0/2522820924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b81057c0 msgr2=0x7f13b819aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b81057c0 0x7f13b819aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 -- 172.21.15.64:0/2522820924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 msgr2=0x7f13b8194b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 0x7f13b8194b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 -- 172.21.15.64:0/2522820924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a80095f0 con 0x7f13b8195080 2023-07-14T18:17:59.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b5d9b700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 0x7f13b8194b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13b6d9d700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13b000ba50 tx=0x7f13b000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b000c9b0 con 0x7f13b8195080 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f13b0014440 con 0x7f13b8195080 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b81a1d00 con 0x7f13b8195080 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b00075b0 con 0x7f13b8195080 2023-07-14T18:17:59.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.274+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b81a21b0 con 0x7f13b8195080 2023-07-14T18:17:59.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.275+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f13b0007710 con 0x7f13b8195080 2023-07-14T18:17:59.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.275+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b8061c70 con 0x7f13b8195080 2023-07-14T18:17:59.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.275+0000 7f13a77fe700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 0x7f13a003b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:17:59.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.275+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13b004aa50 con 0x7f13b8195080 2023-07-14T18:17:59.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.275+0000 7f13b659c700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 0x7f13a003b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:17:59.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.276+0000 7f13b659c700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 0x7f13a003b130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009fd0 tx=0x7f13a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:17:59.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.280+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f13b001b8b0 con 0x7f13b8195080 2023-07-14T18:17:59.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.486+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f13b8195fe0 con 0x7f13b8195080 2023-07-14T18:17:59.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.487+0000 7f13a77fe700 1 -- 172.21.15.64:0/2522820924 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f13b001b2c0 con 0x7f13b8195080 2023-07-14T18:17:59.534 INFO:teuthology.orchestra.run.smithi064.stdout:# minimal ceph.conf for 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:59.534 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2023-07-14T18:17:59.535 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:17:59.535 INFO:teuthology.orchestra.run.smithi064.stdout: mon_host = [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2023-07-14T18:17:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 msgr2=0x7f13a003b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 0x7f13a003b130 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f13a8009fd0 tx=0x7f13a80058e0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 msgr2=0x7f13b8195520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:17:59.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f13b000ba50 tx=0x7f13b000bd60 comp rx=0 tx=0).stop 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 shutdown_connections 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f13b81057c0 0x7f13b819aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f13a0038c80 0x7f13a003b130 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f13b8195080 0x7f13b8195520 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 --2- 172.21.15.64:0/2522820924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13b819b000 0x7f13b8194b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 >> 172.21.15.64:0/2522820924 conn(0x7f13b80743e0 msgr2=0x7f13b8107060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:17:59.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 shutdown_connections 2023-07-14T18:17:59.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:17:59.489+0000 7f13bcad1700 1 -- 172.21.15.64:0/2522820924 wait complete. 2023-07-14T18:17:59.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mon.b calling monitor election 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: Updating smithi064:/etc/ceph/ceph.conf 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mon.b calling monitor election 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mon.c calling monitor election 2023-07-14T18:17:59.559 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mon.a calling monitor election 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: fsmap 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: mgrmap e13: x(active, since 53s) 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:17:59.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: Cluster is now healthy 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: overall HEALTH_OK 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.562 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.562 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:59.562 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:17:59.562 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:17:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:59.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mon.b calling monitor election 2023-07-14T18:17:59.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: Updating smithi064:/etc/ceph/ceph.conf 2023-07-14T18:17:59.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:59.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:59.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mon.b calling monitor election 2023-07-14T18:17:59.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mon.c calling monitor election 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mon.a calling monitor election 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: fsmap 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:59.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: mgrmap e13: x(active, since 53s) 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: Cluster is now healthy 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: overall HEALTH_OK 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:17:59.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:17:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:17:59.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mon.b calling monitor election 2023-07-14T18:17:59.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: Updating smithi064:/etc/ceph/ceph.conf 2023-07-14T18:17:59.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: Updating smithi105:/etc/ceph/ceph.conf 2023-07-14T18:17:59.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: Updating smithi144:/etc/ceph/ceph.conf 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mon.b calling monitor election 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mon.c calling monitor election 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mon.a calling monitor election 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: monmap e3: 3 mons at {a=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} removed_ranks: {} 2023-07-14T18:17:59.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: fsmap 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: mgrmap e13: x(active, since 53s) 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: Cluster is now healthy 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: overall HEALTH_OK 2023-07-14T18:17:59.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:17:59.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:17:59.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:17:59.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:17:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:00.205 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T18:18:00.206 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:18:00.206 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:18:00.237 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:18:00.237 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:18:00.305 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:18:00.305 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:18:00.339 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:18:00.340 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:18:00.408 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:18:00.408 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:18:00.445 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:18:00.445 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:18:00.515 INFO:tasks.cephadm:Adding mgr.x on smithi064 2023-07-14T18:18:00.516 INFO:tasks.cephadm:Adding mgr.y on smithi105 2023-07-14T18:18:00.516 INFO:tasks.cephadm:Adding mgr.z on smithi144 2023-07-14T18:18:00.516 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi064=x;smithi105=y;smithi144=z' 2023-07-14T18:18:00.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: Updating smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:18:00.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: Reconfiguring daemon mon.a on smithi064 2023-07-14T18:18:00.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:18:00.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:00.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:00 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2522820924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:00.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: Updating smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: Reconfiguring daemon mon.a on smithi064 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:00.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:00 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2522820924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:00.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: Updating smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: Updating smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: Updating smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:18:00.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:18:00.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: Reconfiguring daemon mon.a on smithi064 2023-07-14T18:18:00.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:18:00.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:00.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:00 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2522820924' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:01.108 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:18:01.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:01.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:01.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:01.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:01.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:01.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:01.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:01.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.641 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 -- 172.21.15.144:0/1369104230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f600ffdc0 msgr2=0x7f6f601001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:02.642 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 --2- 172.21.15.144:0/1369104230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f600ffdc0 0x7f6f601001d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009910 tx=0x7f6f50009c20 comp rx=0 tx=0).stop 2023-07-14T18:18:02.642 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 -- 172.21.15.144:0/1369104230 shutdown_connections 2023-07-14T18:18:02.642 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 --2- 172.21.15.144:0/1369104230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f600ffdc0 0x7f6f601001d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.642 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 -- 172.21.15.144:0/1369104230 >> 172.21.15.144:0/1369104230 conn(0x7f6f600f9450 msgr2=0x7f6f600fb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:02.642 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 -- 172.21.15.144:0/1369104230 shutdown_connections 2023-07-14T18:18:02.643 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.594+0000 7f6f64b59700 1 -- 172.21.15.144:0/1369104230 wait complete. 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 Processor -- start 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 -- start start 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 0x7f6f6019a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f60195540 0x7f6f6019c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:02.645 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019ce00 con 0x7f6f60195540 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019cf70 con 0x7f6f601950d0 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f64b59700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f6019d130 con 0x7f6f600ffdc0 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f5e59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f5e59c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54708/0 (socket says 172.21.15.144:54708) 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.595+0000 7f6f5e59c700 1 -- 172.21.15.144:0/631576976 learned_addr learned my addr 172.21.15.144:0/631576976 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:18:02.646 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5ed9d700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f60195540 0x7f6f6019c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5dd9b700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 0x7f6f6019a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 -- 172.21.15.144:0/631576976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 msgr2=0x7f6f6019a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 0x7f6f6019a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 -- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f60195540 msgr2=0x7f6f6019c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f60195540 0x7f6f6019c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 -- 172.21.15.144:0/631576976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f500095f0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.647 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5dd9b700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 0x7f6f6019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f5e59c700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000bcf0 tx=0x7f6f5000bd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5001d070 con 0x7f6f600ffdc0 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6f50004db0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f6019d3b0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50022dd0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.648 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.596+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f6019d8d0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.597+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6f5000f460 con 0x7f6f600ffdc0 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.597+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f60104390 con 0x7f6f600ffdc0 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.597+0000 7f6f4f7fe700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 0x7f6f4803b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.597+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6f5004c8b0 con 0x7f6f600ffdc0 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.598+0000 7f6f5dd9b700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 0x7f6f4803b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:02.649 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.598+0000 7f6f5dd9b700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 0x7f6f4803b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009910 tx=0x7f6f54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:02.650 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.602+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6f50020960 con 0x7f6f600ffdc0 2023-07-14T18:18:02.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:18:02.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: Reconfiguring daemon mon.b on smithi105 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:02.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:18:02.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: Reconfiguring daemon mon.b on smithi105 2023-07-14T18:18:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:02.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:02.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:18:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: Reconfiguring daemon mon.b on smithi105 2023-07-14T18:18:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:02.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:18:02.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:18:02.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:02.855 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.808+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi105=y;smithi144=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f600fd8e0 con 0x7f6f48038c30 2023-07-14T18:18:02.861 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.815+0000 7f6f4f7fe700 1 -- 172.21.15.144:0/631576976 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6f600fd8e0 con 0x7f6f48038c30 2023-07-14T18:18:02.862 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mgr update... 2023-07-14T18:18:02.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 msgr2=0x7f6f4803b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:02.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 0x7f6f4803b0e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6f54009910 tx=0x7f6f54008040 comp rx=0 tx=0).stop 2023-07-14T18:18:02.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 msgr2=0x7f6f60194b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:02.864 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000bcf0 tx=0x7f6f5000bd20 comp rx=0 tx=0).stop 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 shutdown_connections 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6f600ffdc0 0x7f6f60194b90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6f48038c30 0x7f6f4803b0e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f60195540 0x7f6f6019c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 --2- 172.21.15.144:0/631576976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f601950d0 0x7f6f6019a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 >> 172.21.15.144:0/631576976 conn(0x7f6f600f9450 msgr2=0x7f6f600fa330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:02.865 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 shutdown_connections 2023-07-14T18:18:02.866 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:02.817+0000 7f6f64b59700 1 -- 172.21.15.144:0/631576976 wait complete. 2023-07-14T18:18:03.337 DEBUG:teuthology.orchestra.run.smithi105:mgr.y> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T18:18:03.341 DEBUG:teuthology.orchestra.run.smithi144:mgr.z> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T18:18:03.343 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T18:18:03.344 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:18:03.344 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:18:03.368 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:18:03.369 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2023-07-14T18:18:03.372 INFO:journalctl@ceph.mgr.z.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:18:03.375 INFO:journalctl@ceph.mgr.y.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:18:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:18:03.430 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 384032 Links: 1 2023-07-14T18:18:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:03.430 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:03.431 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-07-14 18:17:16.129413243 +0000 2023-07-14T18:18:03.431 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-07-14 18:09:35.058464724 +0000 2023-07-14T18:18:03.431 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-07-14 18:09:35.058464724 +0000 2023-07-14T18:18:03.431 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-07-14T18:18:03.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:18:03.500 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-07-14T18:18:03.501 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-07-14T18:18:03.501 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000253102 s, 2.0 MB/s 2023-07-14T18:18:03.502 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:18:03.563 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2023-07-14T18:18:03.607 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:18:03.607 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: Reconfiguring daemon mon.c on smithi144 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:03.608 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.609 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:03.609 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:03.609 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:03.609 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.621 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 380578 Links: 1 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-07-14 18:17:16.129413243 +0000 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-07-14 18:09:35.613454181 +0000 2023-07-14T18:18:03.622 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-07-14 18:09:35.613454181 +0000 2023-07-14T18:18:03.623 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-07-14T18:18:03.623 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:18:03.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:18:03.687 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: Reconfiguring daemon mon.c on smithi144 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:03.688 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.689 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:03.689 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:03.689 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:03.689 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.690 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-07-14T18:18:03.690 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-07-14T18:18:03.690 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000222136 s, 2.3 MB/s 2023-07-14T18:18:03.691 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:18:03.750 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2023-07-14T18:18:03.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:18:03.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: Reconfiguring daemon mon.c on smithi144 2023-07-14T18:18:03.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:03.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:03.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:03.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:03.809 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:18:03.809 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 379890 Links: 1 2023-07-14T18:18:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:03.809 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:03.810 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-07-14 18:17:16.129413243 +0000 2023-07-14T18:18:03.810 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-07-14 18:09:36.133444303 +0000 2023-07-14T18:18:03.810 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-07-14 18:09:36.133444303 +0000 2023-07-14T18:18:03.810 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-07-14T18:18:03.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:18:03.875 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-07-14T18:18:03.876 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-07-14T18:18:03.876 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000235778 s, 2.2 MB/s 2023-07-14T18:18:03.877 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:18:03.936 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2023-07-14T18:18:03.993 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 6h/6d Inode: 377706 Links: 1 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2023-07-14 18:17:16.129413243 +0000 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2023-07-14 18:09:36.650434483 +0000 2023-07-14T18:18:03.994 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2023-07-14 18:09:36.650434483 +0000 2023-07-14T18:18:03.995 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2023-07-14T18:18:03.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:18:04.060 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2023-07-14T18:18:04.060 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2023-07-14T18:18:04.060 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000188449 s, 2.7 MB/s 2023-07-14T18:18:04.061 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:18:04.119 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2023-07-14T18:18:04.119 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:18:04.140 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:18:04.140 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2023-07-14T18:18:04.199 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 379354 Links: 1 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-07-14 18:17:50.838890434 +0000 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-07-14 18:08:48.621442936 +0000 2023-07-14T18:18:04.200 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-07-14 18:08:48.621442936 +0000 2023-07-14T18:18:04.201 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-07-14T18:18:04.201 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:18:04.268 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-07-14T18:18:04.269 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-07-14T18:18:04.269 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000270738 s, 1.9 MB/s 2023-07-14T18:18:04.270 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:18:04.329 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2023-07-14T18:18:04.387 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:18:04.388 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:04.388 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 379392 Links: 1 2023-07-14T18:18:04.389 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:04.389 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:04.389 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-07-14 18:17:50.838890434 +0000 2023-07-14T18:18:04.389 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-07-14 18:08:49.126433413 +0000 2023-07-14T18:18:04.390 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-07-14 18:08:49.126433413 +0000 2023-07-14T18:18:04.390 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-07-14T18:18:04.390 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:18:04.454 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-07-14T18:18:04.455 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-07-14T18:18:04.455 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000250901 s, 2.0 MB/s 2023-07-14T18:18:04.455 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:18:04.514 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2023-07-14T18:18:04.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:04 smithi105 ceph-mon[98013]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi105=y;smithi144=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:04.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:04 smithi105 ceph-mon[98013]: Saving service mgr spec with placement smithi064=x;smithi105=y;smithi144=z;count:3 2023-07-14T18:18:04.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:04 smithi105 ceph-mon[98013]: Deploying daemon mgr.z on smithi144 2023-07-14T18:18:04.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:04 smithi105 ceph-mon[98013]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:04.571 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:18:04.571 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:04.571 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 380142 Links: 1 2023-07-14T18:18:04.571 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:04.572 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:04.572 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-07-14 18:17:50.838890434 +0000 2023-07-14T18:18:04.572 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-07-14 18:08:49.634423833 +0000 2023-07-14T18:18:04.572 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-07-14 18:08:49.634423833 +0000 2023-07-14T18:18:04.572 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-07-14T18:18:04.573 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:18:04.637 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-07-14T18:18:04.637 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-07-14T18:18:04.637 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000239235 s, 2.1 MB/s 2023-07-14T18:18:04.638 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:18:04.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:04 smithi144 ceph-mon[98417]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi105=y;smithi144=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:04.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:04 smithi144 ceph-mon[98417]: Saving service mgr spec with placement smithi064=x;smithi105=y;smithi144=z;count:3 2023-07-14T18:18:04.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:04 smithi144 ceph-mon[98417]: Deploying daemon mgr.z on smithi144 2023-07-14T18:18:04.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:04 smithi144 ceph-mon[98417]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:04.678 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:04 smithi144 systemd[1]: Starting Ceph mgr.z for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:18:04.696 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2023-07-14T18:18:04.753 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 6h/6d Inode: 382276 Links: 1 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2023-07-14 18:17:50.838890434 +0000 2023-07-14T18:18:04.754 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2023-07-14 18:08:50.137414348 +0000 2023-07-14T18:18:04.755 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2023-07-14 18:08:50.137414348 +0000 2023-07-14T18:18:04.755 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: - 2023-07-14T18:18:04.755 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:18:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2023-07-14T18:18:04.818 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2023-07-14T18:18:04.819 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000229861 s, 2.2 MB/s 2023-07-14T18:18:04.819 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:18:04.836 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:04 smithi064 ceph-mon[96720]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi064=x;smithi105=y;smithi144=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:04.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:04 smithi064 ceph-mon[96720]: Saving service mgr spec with placement smithi064=x;smithi105=y;smithi144=z;count:3 2023-07-14T18:18:04.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:04 smithi064 ceph-mon[96720]: Deploying daemon mgr.z on smithi144 2023-07-14T18:18:04.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:04 smithi064 ceph-mon[96720]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:04.878 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2023-07-14T18:18:04.879 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:18:04.897 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:18:04.898 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2023-07-14T18:18:04.955 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:18:04.955 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 374177 Links: 1 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-07-14 18:18:04.388745337 +0000 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-07-14 18:08:46.318113216 +0000 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-07-14 18:08:46.318113216 +0000 2023-07-14T18:18:04.956 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2023-07-14T18:18:04.957 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:18:05.026 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-07-14T18:18:05.026 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-07-14T18:18:05.027 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000228609 s, 2.2 MB/s 2023-07-14T18:18:05.028 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:18:05.058 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:04 smithi144 podman[100277]: 2023-07-14T18:18:05.077 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2023-07-14T18:18:05.134 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:18:05.134 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:05.134 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 374334 Links: 1 2023-07-14T18:18:05.134 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:05.134 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:05.135 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-07-14 18:18:04.388745337 +0000 2023-07-14T18:18:05.135 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-07-14 18:08:46.829105184 +0000 2023-07-14T18:18:05.135 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-07-14 18:08:46.829105184 +0000 2023-07-14T18:18:05.135 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2023-07-14T18:18:05.135 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:18:05.199 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-07-14T18:18:05.199 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-07-14T18:18:05.199 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000249287 s, 2.1 MB/s 2023-07-14T18:18:05.200 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:18:05.257 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2023-07-14T18:18:05.314 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 382569 Links: 1 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-07-14 18:18:04.388745337 +0000 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-07-14 18:08:47.506094542 +0000 2023-07-14T18:18:05.315 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-07-14 18:08:47.506094542 +0000 2023-07-14T18:18:05.316 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2023-07-14T18:18:05.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:18:05.359 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 bash[100277]: a6a0f3806edb5740eee8177de4f7865e6c676ab23945a13d26af3a99581e752d 2023-07-14T18:18:05.360 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-mgr[100376]: -- 172.21.15.144:0/2062725566 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dcde5671e0 con 0x55dcde53c800 2023-07-14T18:18:05.360 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 systemd[1]: Started Ceph mgr.z for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:18:05.360 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.138+0000 7f7111f1b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:18:05.360 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.220+0000 7f7111f1b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:18:05.360 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.312+0000 7f7111f1b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:18:05.380 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-07-14T18:18:05.380 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-07-14T18:18:05.381 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000242466 s, 2.1 MB/s 2023-07-14T18:18:05.381 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:18:05.439 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2023-07-14T18:18:05.496 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 6h/6d Inode: 376065 Links: 1 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2023-07-14 18:18:04.388745337 +0000 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2023-07-14 18:08:48.011086605 +0000 2023-07-14T18:18:05.497 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2023-07-14 18:08:48.011086605 +0000 2023-07-14T18:18:05.498 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: - 2023-07-14T18:18:05.498 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:18:05.562 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2023-07-14T18:18:05.563 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2023-07-14T18:18:05.563 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000220312 s, 2.3 MB/s 2023-07-14T18:18:05.564 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:18:05.621 INFO:tasks.cephadm:Deploying osd.0 on smithi064 with /dev/vg_nvme/lv_4... 2023-07-14T18:18:05.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:18:05.635 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.412+0000 7f7111f1b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:18:05.927 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.588+0000 7f7111f1b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:18:06.195 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:18:06.260 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.895+0000 7f7111f1b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:18:06.260 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:05 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:05.983+0000 7f7111f1b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:18:06.260 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:06.119+0000 7f7111f1b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:18:06.260 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:06.261 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: Deploying daemon mgr.y on smithi105 2023-07-14T18:18:06.262 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-mon[98417]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:06.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:06.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: Deploying daemon mgr.y on smithi105 2023-07-14T18:18:06.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:06 smithi105 ceph-mon[98013]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:06.404 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:06.405 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: Deploying daemon mgr.y on smithi105 2023-07-14T18:18:06.406 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:06 smithi064 ceph-mon[96720]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:06.676 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:06.213+0000 7f7111f1b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:18:06.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:06 smithi105 systemd[1]: Starting Ceph mgr.y for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:18:06.927 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:06 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:06.629+0000 7f7111f1b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:18:07.194 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:06 smithi105 podman[99447]: 2023-07-14T18:18:07.194 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-mgr[99487]: -- 172.21.15.105:0/3563092384 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5622040551e0 con 0x56220402b000 2023-07-14T18:18:07.195 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 bash[99447]: 9bc5d10f7588bcabecacbf7e3cdbfcd8d9e33a1ba98f16277385dcf345feb1c0 2023-07-14T18:18:07.516 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 systemd[1]: Started Ceph mgr.y for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:18:07.516 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.196+0000 7f633f7d3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:18:07.516 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.277+0000 7f633f7d3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:18:07.516 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.371+0000 7f633f7d3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:18:07.516 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.468+0000 7f633f7d3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:18:07.594 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:18:07.594 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:18:07.594 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2023-07-14T18:18:07.594 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2023-07-14T18:18:07.594 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111292 s, 942 MB/s 2023-07-14T18:18:07.595 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2023-07-14T18:18:07.677 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:07 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:07.201+0000 7f7111f1b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:18:07.677 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:07 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:07.373+0000 7f7111f1b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:18:07.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.646+0000 7f633f7d3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:18:07.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2023-07-14T18:18:08.306 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:07 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:07.965+0000 7f633f7d3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:18:08.306 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:08.069+0000 7f633f7d3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:18:08.306 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:08.202+0000 7f633f7d3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-mon[98013]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:08.507 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.507 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.507 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.508 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.508 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:08.508 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-mon[98417]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:08.508 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:08.180+0000 7f7111f1b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:18:08.508 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:08.274+0000 7f7111f1b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:18:08.511 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:08.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:08 smithi064 ceph-mon[96720]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:08.798 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:08.459+0000 7f7111f1b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:18:08.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:08.301+0000 7f633f7d3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:18:09.177 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:08.751+0000 7f7111f1b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:18:09.178 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:08 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:08.996+0000 7f7111f1b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:18:09.306 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:08 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:08.761+0000 7f633f7d3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:18:09.453 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:09.147+0000 7f7111f1b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:18:09.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 -- 172.21.15.64:0/1994228835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2001007c0 msgr2=0x7ff200102ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 --2- 172.21.15.64:0/1994228835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2001007c0 0x7ff200102ba0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff1f0009990 tx=0x7ff1f0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 -- 172.21.15.64:0/1994228835 shutdown_connections 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 --2- 172.21.15.64:0/1994228835 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2001007c0 0x7ff200102ba0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 --2- 172.21.15.64:0/1994228835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff200105a00 0x7ff200107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 --2- 172.21.15.64:0/1994228835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001030e0 0x7ff2001054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 -- 172.21.15.64:0/1994228835 >> 172.21.15.64:0/1994228835 conn(0x7ff2000fa1e0 msgr2=0x7ff2000fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 -- 172.21.15.64:0/1994228835 shutdown_connections 2023-07-14T18:18:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 -- 172.21.15.64:0/1994228835 wait complete. 2023-07-14T18:18:09.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.638+0000 7ff205dce700 1 Processor -- start 2023-07-14T18:18:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 -- start start 2023-07-14T18:18:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 0x7ff200100120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 0x7ff2000fe770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff200105a00 0x7ff2000fecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:09.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000ff430 con 0x7ff2001030e0 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2000ff5a0 con 0x7ff2001007c0 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff205dce700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20019f980 con 0x7ff200105a00 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 0x7ff2000fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 0x7ff200100120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1feffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 0x7ff2000fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:45950/0 (socket says 172.21.15.64:45950) 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1feffd700 1 -- 172.21.15.64:0/1814081520 learned_addr learned my addr 172.21.15.64:0/1814081520 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1fffff700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff200105a00 0x7ff2000fecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:09.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 -- 172.21.15.64:0/1814081520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff200105a00 msgr2=0x7ff2000fecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff200105a00 0x7ff2000fecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 -- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 msgr2=0x7ff2000fe770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 0x7ff2000fe770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 -- 172.21.15.64:0/1814081520 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009520 con 0x7ff2001007c0 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.639+0000 7ff1ff7fe700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 0x7ff200100120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098e0 tx=0x7ff1f0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f001d070 con 0x7ff2001007c0 2023-07-14T18:18:09.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f00095f0 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff1f000bdb0 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0022d00 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20019fe80 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.640+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7ff1f000f460 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.641+0000 7ff1fcff9700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 0x7ff1e803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.641+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2000fbd50 con 0x7ff2001007c0 2023-07-14T18:18:09.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.641+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff1f004c810 con 0x7ff2001007c0 2023-07-14T18:18:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.641+0000 7ff1feffd700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 0x7ff1e803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.641+0000 7ff1feffd700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 0x7ff1e803ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009ef0 tx=0x7ff1f4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:09.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.646+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff1f00167b0 con 0x7ff2001007c0 2023-07-14T18:18:09.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:09.349+0000 7f633f7d3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:18:09.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:09.516+0000 7f633f7d3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:18:09.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:09.859+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff200060a80 con 0x7ff1e80389e0 2023-07-14T18:18:09.927 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:09.406+0000 7f7111f1b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:18:10.081 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.082 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: Reconfiguring daemon mgr.x on smithi064 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:10.083 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:10.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:10.084 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.306 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: Reconfiguring daemon mgr.x on smithi064 2023-07-14T18:18:10.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:10.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:10.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:10.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.309 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:10 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:10.130+0000 7f7111f1b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:18:10.309 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.309 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.309 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.309 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.310 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: Reconfiguring daemon mgr.x on smithi064 2023-07-14T18:18:10.311 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:10.311 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:10.311 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:10.311 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:10.582 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:10 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:10.276+0000 7f633f7d3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:18:10.582 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:10 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:10.365+0000 7f633f7d3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:18:10.614 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:10 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:10.260+0000 7f7111f1b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:18:10.614 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:10 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:10.453+0000 7f7111f1b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:18:10.927 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:10 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:10.567+0000 7f7111f1b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:18:10.927 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:10 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:10.781+0000 7f7111f1b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:18:11.237 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:10 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:10.535+0000 7f633f7d3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:18:11.237 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:10 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:10.808+0000 7f633f7d3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:18:11.237 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:11.045+0000 7f633f7d3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:18:11.237 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:11.190+0000 7f633f7d3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:18:11.497 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:11.497 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.497 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:11.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:11.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:11.498 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:11.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:11.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:11.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:11.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:11.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:11.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:11.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:11 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:11.450+0000 7f633f7d3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:18:12.177 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:18:11 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z[100293]: 2023-07-14T18:18:11.635+0000 7f7111f1b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:18:12.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:12.299+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7ff1f0022470 con 0x7ff2001007c0 2023-07-14T18:18:12.517 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:12.148+0000 7f633f7d3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:12.280+0000 7f633f7d3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: Standby manager daemon z started 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:12.518 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:18:12.519 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:12.519 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]: dispatch 2023-07-14T18:18:12.519 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]': finished 2023-07-14T18:18:12.519 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:18:12.519 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:12.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: Standby manager daemon z started 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:12.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]: dispatch 2023-07-14T18:18:12.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]': finished 2023-07-14T18:18:12.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:18:12.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:12.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:12.471+0000 7f633f7d3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:18:12.806 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:12.585+0000 7f633f7d3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: Standby manager daemon z started 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:18:12.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.144:0/4034116347' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]: dispatch 2023-07-14T18:18:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2545803600' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a183b85a-fd4d-4bc0-ad4f-40a7e3761093"}]': finished 2023-07-14T18:18:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:18:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:13.306 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:12.800+0000 7f633f7d3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:18:13.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:13 smithi144 ceph-mon[98417]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T18:18:13.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:18:13.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:13 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1439333792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:13 smithi105 ceph-mon[98013]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T18:18:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:18:13.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:13 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1439333792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:13.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:13 smithi064 ceph-mon[96720]: mgrmap e14: x(active, since 66s), standbys: z 2023-07-14T18:18:13.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:18:13.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:13 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1439333792' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:14.056 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:18:13 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y[99463]: 2023-07-14T18:18:13.645+0000 7f633f7d3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:18:14.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:14.522+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mon.2 v2:172.21.15.105:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7ff1f00675e0 con 0x7ff2001007c0 2023-07-14T18:18:14.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:14.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: Standby manager daemon y started 2023-07-14T18:18:14.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:18:14.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:14.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:18:14.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:14 smithi144 ceph-mon[98417]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:14.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:14.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: Standby manager daemon y started 2023-07-14T18:18:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:18:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:18:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:14 smithi105 ceph-mon[98013]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:14.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:14.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: Standby manager daemon y started 2023-07-14T18:18:14.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:18:14.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:18:14.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:18:14.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:14 smithi064 ceph-mon[96720]: from='mgr.? 172.21.15.105:0/2672574641' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:18:15.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:15 smithi064 ceph-mon[96720]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-14T18:18:15.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:18:15.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:15 smithi064 ceph-mon[96720]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:15.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:15 smithi144 ceph-mon[98417]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-14T18:18:15.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:18:15.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:15 smithi144 ceph-mon[98417]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:16.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:15 smithi105 ceph-mon[98013]: mgrmap e15: x(active, since 69s), standbys: z, y 2023-07-14T18:18:16.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:18:16.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:15 smithi105 ceph-mon[98013]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:17.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:17 smithi105 ceph-mon[98013]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:17.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:17 smithi064 ceph-mon[96720]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:17.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:17 smithi144 ceph-mon[98417]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:19.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:18:19.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:19.739 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:19 smithi064 ceph-mon[96720]: Deploying daemon osd.0 on smithi064 2023-07-14T18:18:19.740 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:19 smithi064 ceph-mon[96720]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:19.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:18:19.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:19.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:19 smithi144 ceph-mon[98417]: Deploying daemon osd.0 on smithi064 2023-07-14T18:18:19.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:19 smithi144 ceph-mon[98417]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:20.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:18:20.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:20.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:19 smithi105 ceph-mon[98013]: Deploying daemon osd.0 on smithi064 2023-07-14T18:18:20.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:19 smithi105 ceph-mon[98013]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:21.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:21 smithi105 ceph-mon[98013]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:21.819 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:21 smithi064 ceph-mon[96720]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:21.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:21 smithi144 ceph-mon[98417]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:24.505 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:24 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.506 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:24 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.506 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:24 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:24.506 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:24 smithi064 ceph-mon[96720]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:24.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:24 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:24 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:24 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:24.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:24 smithi105 ceph-mon[98013]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:24.677 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:24 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:24 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:24.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:24 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:24.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:24 smithi144 ceph-mon[98417]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:26.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.163+0000 7ff1fcff9700 1 -- 172.21.15.64:0/1814081520 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff200060a80 con 0x7ff1e80389e0 2023-07-14T18:18:26.210 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 0 on host 'smithi064' 2023-07-14T18:18:26.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 msgr2=0x7ff1e803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:26.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 0x7ff1e803ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009ef0 tx=0x7ff1f4009260 comp rx=0 tx=0).stop 2023-07-14T18:18:26.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 msgr2=0x7ff200100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:26.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 0x7ff200100120 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00098e0 tx=0x7ff1f0004730 comp rx=0 tx=0).stop 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 shutdown_connections 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff200105a00 0x7ff2000fecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff1e80389e0 0x7ff1e803ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2001007c0 0x7ff200100120 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 --2- 172.21.15.64:0/1814081520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2001030e0 0x7ff2000fe770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 >> 172.21.15.64:0/1814081520 conn(0x7ff2000fa1e0 msgr2=0x7ff200101420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:26.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 shutdown_connections 2023-07-14T18:18:26.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:18:26.166+0000 7ff205dce700 1 -- 172.21.15.64:0/1814081520 wait complete. 2023-07-14T18:18:26.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:26 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:26.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:26 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:26.600 DEBUG:teuthology.orchestra.run.smithi064:osd.0> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T18:18:26.603 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_4... 2023-07-14T18:18:26.603 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:18:26.633 INFO:journalctl@ceph.osd.0.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:18:26.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:26.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:26.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:26.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:26.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:26 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:27.162 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:18:27.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:27 smithi064 ceph-mon[96720]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:18:27.397 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:18:27 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:18:27.189+0000 7ff7f6cdd640 -1 osd.0 0 log_to_monitors true 2023-07-14T18:18:27.552 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:27 smithi105 ceph-mon[98013]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:18:27.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:27 smithi144 ceph-mon[98417]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:18:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:28 smithi105 ceph-mon[98013]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:18:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:28 smithi105 ceph-mon[98013]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:18:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:28 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:28.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:28 smithi105 ceph-mon[98013]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:18:28.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:28 smithi105 ceph-mon[98013]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:28.562 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:18:28.258+0000 7ff7ec355700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T18:18:28.562 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:18:28.261+0000 7ff7e814b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:18:28.563 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-mon[96720]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:18:28.563 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-mon[96720]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:18:28.563 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:28.563 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-mon[96720]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:18:28.563 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:28 smithi064 ceph-mon[96720]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:28 smithi144 ceph-mon[98417]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:18:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:28 smithi144 ceph-mon[98417]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:18:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:28 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:28 smithi144 ceph-mon[98417]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:18:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:28 smithi144 ceph-mon[98417]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:28.752 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:18:28.752 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:18:28.753 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2023-07-14T18:18:28.753 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2023-07-14T18:18:28.753 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111024 s, 944 MB/s 2023-07-14T18:18:28.753 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2023-07-14T18:18:29.203 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2023-07-14T18:18:29.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:18:29.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:18:29.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:29.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:29.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:29.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:29 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:18:29.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:29.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:29.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:29.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:29 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:18:29.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:18:29.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:29.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:29 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:29.754 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:18:30.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:18:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:18:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi064 2023-07-14T18:18:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi064 to 17168M 2023-07-14T18:18:30.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] boot 2023-07-14T18:18:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:18:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:30 smithi105 ceph-mon[98013]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:18:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:18:30.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi064 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi064 to 17168M 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] boot 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:30 smithi064 ceph-mon[96720]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi064 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi064 to 17168M 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: osd.0 [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] boot 2023-07-14T18:18:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:18:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:18:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:30 smithi144 ceph-mon[98417]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:18:30.947 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 -- 172.21.15.105:0/1742906803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b80fee50 msgr2=0x7f75b80ff2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:30.948 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 --2- 172.21.15.105:0/1742906803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b80fee50 0x7f75b80ff2a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f75a0009990 tx=0x7f75a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.948 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 -- 172.21.15.105:0/1742906803 shutdown_connections 2023-07-14T18:18:30.948 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 --2- 172.21.15.105:0/1742906803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b81001a0 0x7f75b81005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.948 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 --2- 172.21.15.105:0/1742906803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 0x7f75b81070b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.948 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 --2- 172.21.15.105:0/1742906803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b80fee50 0x7f75b80ff2a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 -- 172.21.15.105:0/1742906803 >> 172.21.15.105:0/1742906803 conn(0x7f75b80f9990 msgr2=0x7f75b80fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.899+0000 7f75be467700 1 -- 172.21.15.105:0/1742906803 shutdown_connections 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 -- 172.21.15.105:0/1742906803 wait complete. 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 Processor -- start 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 -- start start 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b80fee50 0x7f75b80fe8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:30.949 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 0x7f75b819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b819d540 con 0x7f75b80ff7e0 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b819d6b0 con 0x7f75b81001a0 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75be467700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b819d870 con 0x7f75b80fee50 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75bca04700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75bca04700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57120/0 (socket says 172.21.15.105:57120) 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75affff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 0x7f75b819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:30.950 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75bca04700 1 -- 172.21.15.105:0/654085782 learned_addr learned my addr 172.21.15.105:0/654085782 (peer_addr_for_me v2:172.21.15.105:0/0) 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.900+0000 7f75b7fff700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b80fee50 0x7f75b80fe8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 -- 172.21.15.105:0/654085782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b80fee50 msgr2=0x7f75b80fe8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b80fee50 0x7f75b80fe8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 -- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 msgr2=0x7f75b819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 0x7f75b819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 -- 172.21.15.105:0/654085782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a00095f0 con 0x7f75b81001a0 2023-07-14T18:18:30.951 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75bca04700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75b000ada0 tx=0x7f75b000c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:30.952 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b000f070 con 0x7f75b81001a0 2023-07-14T18:18:30.952 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b819db50 con 0x7f75b81001a0 2023-07-14T18:18:30.952 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f75b0007b30 con 0x7f75b81001a0 2023-07-14T18:18:30.953 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b0005280 con 0x7f75b81001a0 2023-07-14T18:18:30.953 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.901+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b819e0f0 con 0x7f75b81001a0 2023-07-14T18:18:30.953 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.902+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b8106290 con 0x7f75b81001a0 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.903+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f75b000a090 con 0x7f75b81001a0 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.904+0000 7f75b5ffb700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 0x7f75a40a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.904+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f75b00ca930 con 0x7f75b81001a0 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.904+0000 7f75b7fff700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 0x7f75a40a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.904+0000 7f75b7fff700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 0x7f75a40a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75b8067ec0 tx=0x7f75a8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:30.954 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:30.907+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f75b0095b80 con 0x7f75b81001a0 2023-07-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:31.098+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f75b8060a80 con 0x7f75a40a10c0 2023-07-14T18:18:31.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:31 smithi064 ceph-mon[96720]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:18:31.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:31 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:31.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:31 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:31.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:31 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:31.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:31 smithi144 ceph-mon[98417]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:18:31.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:31 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:31.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:31 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:31.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:31 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:31.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:31 smithi105 ceph-mon[98013]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:18:31.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:31 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:31.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:31 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:31.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:31 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:32.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:32 smithi105 ceph-mon[98013]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:32.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:32 smithi105 ceph-mon[98013]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:32.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:32 smithi064 ceph-mon[96720]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:32.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:32 smithi064 ceph-mon[96720]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:32.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:32 smithi144 ceph-mon[98417]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:32.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:32 smithi144 ceph-mon[98417]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:33.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/1851332520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]': finished 2023-07-14T18:18:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:18:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:33.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:33.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/1851332520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]': finished 2023-07-14T18:18:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:18:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:33.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:33.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/1851332520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]: dispatch 2023-07-14T18:18:33.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "195865a8-38c0-48ce-afbd-447f46cc517a"}]': finished 2023-07-14T18:18:33.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:18:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:34.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:34 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/3060642574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:34.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:34 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/3060642574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:35.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:34 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/3060642574' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:35.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:35 smithi064 ceph-mon[96720]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:35.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:35 smithi144 ceph-mon[98417]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:36.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:35 smithi105 ceph-mon[98013]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:37.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:37 smithi105 ceph-mon[98013]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:37.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:37 smithi064 ceph-mon[96720]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:37.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:37 smithi144 ceph-mon[98417]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:39.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:39 smithi105 ceph-mon[98013]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:39.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:39 smithi064 ceph-mon[96720]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:39.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:39 smithi144 ceph-mon[98417]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:40.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:18:40.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:40.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:40 smithi105 ceph-mon[98013]: Deploying daemon osd.1 on smithi105 2023-07-14T18:18:40.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:18:40.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:40.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:40 smithi064 ceph-mon[96720]: Deploying daemon osd.1 on smithi105 2023-07-14T18:18:40.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:18:40.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:40.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:40 smithi144 ceph-mon[98417]: Deploying daemon osd.1 on smithi105 2023-07-14T18:18:41.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:41 smithi105 ceph-mon[98013]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:41.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:41 smithi144 ceph-mon[98417]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:42.087 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:41 smithi064 ceph-mon[96720]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:43.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:43 smithi105 ceph-mon[98013]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:43.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:43 smithi064 ceph-mon[96720]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:43.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:43 smithi144 ceph-mon[98417]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:45.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:45 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:45 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:45 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:45.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:45 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:45 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:45 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:45.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:45 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:45 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:45.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:45 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:46.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:46 smithi105 ceph-mon[98013]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:46.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:46.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:46.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:46 smithi064 ceph-mon[96720]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:46.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:46.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:46.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:46 smithi144 ceph-mon[98417]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:46.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:46.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.061 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.013+0000 7f75b5ffb700 1 -- 172.21.15.105:0/654085782 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f75b8060a80 con 0x7f75a40a10c0 2023-07-14T18:18:47.061 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2023-07-14T18:18:47.063 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 msgr2=0x7f75a40a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:47.063 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 0x7f75a40a3570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f75b8067ec0 tx=0x7f75a8009190 comp rx=0 tx=0).stop 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 msgr2=0x7f75b819cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75b000ada0 tx=0x7f75b000c330 comp rx=0 tx=0).stop 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 shutdown_connections 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f75b80fee50 0x7f75b80fe8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f75a40a10c0 0x7f75a40a3570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:47.064 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f75b81001a0 0x7f75b819cda0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:47.065 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 --2- 172.21.15.105:0/654085782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f75b80ff7e0 0x7f75b819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:47.065 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.015+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 >> 172.21.15.105:0/654085782 conn(0x7f75b80f9990 msgr2=0x7f75b81033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:47.065 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.016+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 shutdown_connections 2023-07-14T18:18:47.065 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:18:47.016+0000 7f75be467700 1 -- 172.21.15.105:0/654085782 wait complete. 2023-07-14T18:18:47.438 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T18:18:47.441 INFO:tasks.cephadm:Deploying osd.2 on smithi144 with /dev/vg_nvme/lv_4... 2023-07-14T18:18:47.442 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:18:47.467 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:47.467 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:47.467 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.468 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.468 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:47.468 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:47 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.474 INFO:journalctl@ceph.osd.1.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:18:47.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:47.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:47.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:47.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:47 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:47.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:47.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:47.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:18:47.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:47 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:48.004 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:18:48.492 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:18:48 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:18:48.165+0000 7f83a1293640 -1 osd.1 0 log_to_monitors true 2023-07-14T18:18:48.492 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:48 smithi105 ceph-mon[98013]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:48.492 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:48 smithi105 ceph-mon[98013]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:48.493 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:48 smithi105 ceph-mon[98013]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:48.535 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:48 smithi144 ceph-mon[98417]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:48.536 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:48 smithi144 ceph-mon[98417]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:48.536 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:48 smithi144 ceph-mon[98417]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:48.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:48 smithi064 ceph-mon[96720]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:48.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:48 smithi064 ceph-mon[96720]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:48.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:48 smithi064 ceph-mon[96720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:18:49.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:49 smithi144 ceph-mon[98417]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:18:49.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:49 smithi144 ceph-mon[98417]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:18:49.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:49.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:49 smithi144 ceph-mon[98417]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:49 smithi144 ceph-mon[98417]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:18:49.231+0000 7f839690b700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T18:18:49.555 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:18:49.234+0000 7f83906fd700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:18:49.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-mon[98013]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:18:49.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-mon[98013]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:18:49.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:49.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-mon[98013]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:49 smithi105 ceph-mon[98013]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:49 smithi064 ceph-mon[96720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:18:49.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:49 smithi064 ceph-mon[96720]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:18:49.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:49.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:49 smithi064 ceph-mon[96720]: from='osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:49 smithi064 ceph-mon[96720]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:18:49.616 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:18:49.616 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:18:49.616 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2023-07-14T18:18:49.617 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2023-07-14T18:18:49.617 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129473 s, 810 MB/s 2023-07-14T18:18:49.617 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2023-07-14T18:18:50.144 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi144:vg_nvme/lv_4 2023-07-14T18:18:50.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:18:50.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:18:50.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:50.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:50.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:18:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:18:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:50.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:50.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:18:50.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:50.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:18:50.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:18:50.696 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:18:51.542 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:18:51.542 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:18:51.542 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi105 2023-07-14T18:18:51.543 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi105 to 17168M 2023-07-14T18:18:51.543 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] boot 2023-07-14T18:18:51.543 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:18:51.543 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:51.556 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi105 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi105 to 17168M 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] boot 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:18:51.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:51.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi105 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi105 to 17168M 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: osd.1 [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] boot 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:18:51.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:18:52.103 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 -- 172.21.15.144:0/1617933371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 msgr2=0x7f19d40fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:52.104 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/1617933371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d40fefc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f19c4009990 tx=0x7f19c4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.104 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 -- 172.21.15.144:0/1617933371 shutdown_connections 2023-07-14T18:18:52.104 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/1617933371 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d40fefc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.104 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/1617933371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d4100740 0x7f19d4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.104 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/1617933371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d40ffdb0 0x7f19d4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 -- 172.21.15.144:0/1617933371 >> 172.21.15.144:0/1617933371 conn(0x7f19d40fa1e0 msgr2=0x7f19d40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 -- 172.21.15.144:0/1617933371 shutdown_connections 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.055+0000 7f19d8ff3700 1 -- 172.21.15.144:0/1617933371 wait complete. 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 Processor -- start 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 -- start start 2023-07-14T18:18:52.105 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 0x7f19d41980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 0x7f19d419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d419f720 con 0x7f19d40ffdb0 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d419f860 con 0x7f19d4100740 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d8ff3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d419f9f0 con 0x7f19d40febb0 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54740/0 (socket says 172.21.15.144:54740) 2023-07-14T18:18:52.106 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 -- 172.21.15.144:0/3903556764 learned_addr learned my addr 172.21.15.144:0/3903556764 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d1d9b700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 0x7f19d41980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d2d9d700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 0x7f19d419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 -- 172.21.15.144:0/3903556764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 msgr2=0x7f19d419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 0x7f19d419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 -- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 msgr2=0x7f19d41980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:18:52.107 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 0x7f19d41980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 -- 172.21.15.144:0/3903556764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c40095f0 con 0x7f19d40febb0 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d2d9d700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 0x7f19d419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d1d9b700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 0x7f19d41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.056+0000 7f19d259c700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004760 tx=0x7f19c40038d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.057+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c401d070 con 0x7f19d40febb0 2023-07-14T18:18:52.108 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.057+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f19c4005030 con 0x7f19d40febb0 2023-07-14T18:18:52.109 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.057+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d419fc70 con 0x7f19d40febb0 2023-07-14T18:18:52.109 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.057+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c4004a60 con 0x7f19d40febb0 2023-07-14T18:18:52.109 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.057+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d41a00e0 con 0x7f19d40febb0 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.058+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d410a2f0 con 0x7f19d40febb0 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.059+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f19c40051a0 con 0x7f19d40febb0 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.059+0000 7f19c37fe700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 0x7f19bc0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.059+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f19c40cc1f0 con 0x7f19d40febb0 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.060+0000 7f19d1d9b700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 0x7f19bc0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:18:52.110 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.060+0000 7f19d1d9b700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 0x7f19bc0a35f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c8005950 tx=0x7f19c8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:18:52.111 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.064+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f19c4097220 con 0x7f19d40febb0 2023-07-14T18:18:52.303 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:18:52.256+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f19d4068870 con 0x7f19bc0a1140 2023-07-14T18:18:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:52 smithi105 ceph-mon[98013]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:18:52.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:52 smithi105 ceph-mon[98013]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:52.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:52 smithi064 ceph-mon[96720]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:18:52.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:52 smithi064 ceph-mon[96720]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:52.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:52 smithi144 ceph-mon[98417]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:18:52.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:52 smithi144 ceph-mon[98417]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:53.418 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:53 smithi144 ceph-mon[98417]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:53.418 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:53.418 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:53.418 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:53.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:53 smithi105 ceph-mon[98013]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:53.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:53.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:53.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:53.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:53 smithi064 ceph-mon[96720]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:18:53.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:18:53.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:18:53.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:18:54.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:54 smithi105 ceph-mon[98013]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:54.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:54 smithi064 ceph-mon[96720]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:54.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:54 smithi144 ceph-mon[98417]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:55 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/1735496705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:55 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:55 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]': finished 2023-07-14T18:18:55.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:55 smithi105 ceph-mon[98013]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:18:55.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:18:55.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:55 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/1735496705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:55 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:55 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]': finished 2023-07-14T18:18:55.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:55 smithi064 ceph-mon[96720]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:18:55.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:18:55.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:55 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/1735496705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:55 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]: dispatch 2023-07-14T18:18:55.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:55 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c631140b-b28e-41b6-9d45-259e15845aea"}]': finished 2023-07-14T18:18:55.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:55 smithi144 ceph-mon[98417]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:18:55.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:18:56.587 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:56 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/3078598521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:56.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:56 smithi064 ceph-mon[96720]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:56.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:56 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/3078598521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:56.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:56 smithi144 ceph-mon[98417]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:56 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/3078598521' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:18:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:56 smithi105 ceph-mon[98013]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:57.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:57 smithi105 ceph-mon[98013]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:57.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:57 smithi064 ceph-mon[96720]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:57.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:57 smithi144 ceph-mon[98417]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:59.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:18:59 smithi064 ceph-mon[96720]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:18:59.892 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:18:59 smithi144 ceph-mon[98417]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:00.056 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:18:59 smithi105 ceph-mon[98013]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:19:01.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:01.806 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:19:01.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:01.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:19:01.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:02 smithi144 ceph-mon[98417]: Deploying daemon osd.2 on smithi144 2023-07-14T18:19:02.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:02 smithi144 ceph-mon[98417]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:02 smithi105 ceph-mon[98013]: Deploying daemon osd.2 on smithi144 2023-07-14T18:19:02.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:02 smithi105 ceph-mon[98013]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:02.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:02 smithi064 ceph-mon[96720]: Deploying daemon osd.2 on smithi144 2023-07-14T18:19:02.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:02 smithi064 ceph-mon[96720]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:03.800 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:03 smithi144 ceph-mon[98417]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:03 smithi105 ceph-mon[98013]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:03.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:03 smithi064 ceph-mon[96720]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:05.778 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:05 smithi144 ceph-mon[98417]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:05.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:05 smithi105 ceph-mon[98013]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:05.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:05 smithi064 ceph-mon[96720]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:07.082 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.083 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.083 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:07.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:07.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:07.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:08.178 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:07 smithi144 ceph-mon[98417]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:08.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:07 smithi105 ceph-mon[98013]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:08.337 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:07 smithi064 ceph-mon[96720]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:09.017 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 2 on host 'smithi144' 2023-07-14T18:19:09.018 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.969+0000 7f19c37fe700 1 -- 172.21.15.144:0/3903556764 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f19d4068870 con 0x7f19bc0a1140 2023-07-14T18:19:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 msgr2=0x7f19bc0a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 0x7f19bc0a35f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c8005950 tx=0x7f19c8005f50 comp rx=0 tx=0).stop 2023-07-14T18:19:09.020 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 msgr2=0x7f19d4197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f19c4004760 tx=0x7f19c40038d0 comp rx=0 tx=0).stop 2023-07-14T18:19:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 shutdown_connections 2023-07-14T18:19:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f19d40febb0 0x7f19d4197b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:09.021 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19bc0a1140 0x7f19bc0a35f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19d4100740 0x7f19d419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 --2- 172.21.15.144:0/3903556764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19d40ffdb0 0x7f19d41980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.971+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 >> 172.21.15.144:0/3903556764 conn(0x7f19d40fa1e0 msgr2=0x7f19d4068160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:09.022 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.972+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 shutdown_connections 2023-07-14T18:19:09.023 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:08.972+0000 7f19d8ff3700 1 -- 172.21.15.144:0/3903556764 wait complete. 2023-07-14T18:19:09.492 DEBUG:teuthology.orchestra.run.smithi144:osd.2> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T18:19:09.494 INFO:tasks.cephadm:Deploying osd.3 on smithi064 with /dev/vg_nvme/lv_3... 2023-07-14T18:19:09.495 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:19:09.526 INFO:journalctl@ceph.osd.2.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:19:09.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:09.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:09 smithi105 ceph-mon[98013]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:09.833 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.833 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.833 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:09.833 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:09.833 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.834 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.834 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:09.834 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.834 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-mon[98417]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:09.837 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:09.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:09.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:09 smithi064 ceph-mon[96720]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:10.071 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:19:10.178 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:09 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:19:09.784+0000 7f222e5dc640 -1 osd.2 0 log_to_monitors true 2023-07-14T18:19:10.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:10 smithi064 ceph-mon[96720]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:10.558 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:10 smithi064 ceph-mon[96720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:10.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:10 smithi105 ceph-mon[98013]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:10.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:10 smithi105 ceph-mon[98013]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:10.880 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:10 smithi144 ceph-mon[98417]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:10.880 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:10 smithi144 ceph-mon[98417]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:19:11.594 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:19:11.595 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:19:11.595 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2023-07-14T18:19:11.595 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2023-07-14T18:19:11.595 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111892 s, 937 MB/s 2023-07-14T18:19:11.595 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2023-07-14T18:19:11.781 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:19:11.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:19:11.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:11.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:11 smithi064 ceph-mon[96720]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:11.803 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:19:11.467+0000 7f2223c54700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T18:19:11.803 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:19:11.471+0000 7f221ba42700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:19:11.803 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:19:11.803 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:19:11.803 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:11.804 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.804 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.804 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:11 smithi144 ceph-mon[98417]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: from='osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:19:11.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:11 smithi105 ceph-mon[98013]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:19:12.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2023-07-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:19:12.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:19:12.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:19:12.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi144 2023-07-14T18:19:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi144 to 17168M 2023-07-14T18:19:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:19:12.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:19:12.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi144 2023-07-14T18:19:12.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi144 to 17168M 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:12.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi144 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:12.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi144 to 17168M 2023-07-14T18:19:12.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:12.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:12.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:12.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:13.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:19:13.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:19:13.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] boot 2023-07-14T18:19:13.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:19:13.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:13.792 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:13 smithi064 ceph-mon[96720]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] boot 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:13.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:13 smithi105 ceph-mon[98013]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:13.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:19:13.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:19:13.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: osd.2 [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] boot 2023-07-14T18:19:13.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:19:13.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:19:13.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:13 smithi144 ceph-mon[98417]: pgmap v72: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- 172.21.15.64:0/3554730977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 msgr2=0x7f193c107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 --2- 172.21.15.64:0/3554730977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c107de0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f193400b0a0 tx=0x7f193400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- 172.21.15.64:0/3554730977 shutdown_connections 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 --2- 172.21.15.64:0/3554730977 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c100740 0x7f193c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 --2- 172.21.15.64:0/3554730977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c107de0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 --2- 172.21.15.64:0/3554730977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f193c103060 0x7f193c105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- 172.21.15.64:0/3554730977 >> 172.21.15.64:0/3554730977 conn(0x7f193c0fa1e0 msgr2=0x7f193c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- 172.21.15.64:0/3554730977 shutdown_connections 2023-07-14T18:19:13.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- 172.21.15.64:0/3554730977 wait complete. 2023-07-14T18:19:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 Processor -- start 2023-07-14T18:19:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.894+0000 7f19427e5700 1 -- start start 2023-07-14T18:19:13.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f193c100740 0x7f193c1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c19d580 con 0x7f193c100740 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c19d6c0 con 0x7f193c105980 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f19427e5700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193c19d850 con 0x7f193c103060 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:13.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f1940d82700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:53682/0 (socket says 172.21.15.64:53682) 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 -- 172.21.15.64:0/1940279387 learned_addr learned my addr 172.21.15.64:0/1940279387 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 -- 172.21.15.64:0/1940279387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 msgr2=0x7f193c19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 -- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f193c100740 msgr2=0x7f193c1959f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f193c100740 0x7f193c1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:13.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f193b7fe700 1 -- 172.21.15.64:0/1940279387 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1934009d00 con 0x7f193c103060 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.895+0000 7f1940d82700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f193b7fe700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9d0 tx=0x7f193000ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930009790 con 0x7f193c103060 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193c19db30 con 0x7f193c103060 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f193000cbd0 con 0x7f193c103060 2023-07-14T18:19:13.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1930018680 con 0x7f193c103060 2023-07-14T18:19:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.896+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f193c19e080 con 0x7f193c103060 2023-07-14T18:19:13.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.897+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193c10b2d0 con 0x7f193c103060 2023-07-14T18:19:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.898+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f193000cd40 con 0x7f193c103060 2023-07-14T18:19:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.899+0000 7f19397fa700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 0x7f19240a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.899+0000 7f193bfff700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 0x7f19240a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.900+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f1930014070 con 0x7f193c103060 2023-07-14T18:19:13.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.900+0000 7f193bfff700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 0x7f19240a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c005950 tx=0x7f192c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:13.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:13.902+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1930096370 con 0x7f193c103060 2023-07-14T18:19:14.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:14.108+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f193c102290 con 0x7f19240a1140 2023-07-14T18:19:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:19:14.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:14.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:14.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:14.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:14.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:14.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:19:14.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:14.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:14.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:14.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:14.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:14.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:19:14.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:14.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:14.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:14.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:15.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:15.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:15 smithi064 ceph-mon[96720]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:19:15.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:15.791 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:15 smithi064 ceph-mon[96720]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:15.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:15.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:15 smithi105 ceph-mon[98013]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:19:15.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:15.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:15 smithi105 ceph-mon[98013]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:15.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:15.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:15 smithi144 ceph-mon[98417]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:19:15.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:19:15.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:15 smithi144 ceph-mon[98417]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:16.338 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112823]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:16.338 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112823]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:16.338 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112823]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:16.339 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112823]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:16.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:16.499+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1930096740 con 0x7f193c103060 2023-07-14T18:19:16.802 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:16.802 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:19:16.802 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]: dispatch 2023-07-14T18:19:16.803 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]': finished 2023-07-14T18:19:16.803 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:19:16.803 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:16.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:16.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:19:16.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]: dispatch 2023-07-14T18:19:16.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]': finished 2023-07-14T18:19:16.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:19:16.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:16.839 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112858]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:16.839 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:16.840 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112858]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:16.840 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:19:16 smithi064 sudo[112858]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:16.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:19:16.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:19:16.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]: dispatch 2023-07-14T18:19:16.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2945274280' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fe2d4b3-b366-4899-8979-87184b007cb8"}]': finished 2023-07-14T18:19:16.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:19:16.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:17.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:16 smithi105 sudo[105262]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:17.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:16 smithi105 sudo[105262]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:17.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:16 smithi105 sudo[105262]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:17.056 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:16 smithi105 sudo[105262]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:17.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:17 smithi105 sudo[105265]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:17.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:17 smithi105 sudo[105265]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:17.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:17 smithi105 sudo[105265]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:17.307 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:19:17 smithi105 sudo[105265]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:17.579 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106467]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:17.580 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:17.580 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106467]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:17.580 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106467]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:17.580 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:17 smithi144 ceph-mon[98417]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:19:17.581 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.581 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.581 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:17 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2133078676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:17.581 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:17 smithi144 ceph-mon[98417]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:17.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:17 smithi105 ceph-mon[98013]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:19:17.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:17 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2133078676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:17.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:17 smithi105 ceph-mon[98013]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:17.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 ceph-mon[96720]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:19:17.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:17.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2133078676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:17.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 ceph-mon[96720]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:17.928 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106470]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:17.929 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:17.929 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106470]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:17.929 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:19:17 smithi144 sudo[106470]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:18.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 sudo[113312]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:18.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 sudo[113312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:18.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:17 smithi064 sudo[113312]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:18.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 sudo[113312]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:18.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 sudo[113317]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:18.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 sudo[113317]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:18.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 sudo[113317]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:18.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 sudo[113317]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:18.993 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106473]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:18.993 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:18.993 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106473]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:18.994 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106473]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:19.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:19.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:19.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:19.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106476]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106476]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 sudo[106476]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:19.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105268]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:19:19.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105268]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:19.488 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105268]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:19.489 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105268]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:19.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105271]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:19:19.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105271]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:19:19.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105271]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:19:19.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 sudo[105271]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:19 smithi105 ceph-mon[98013]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:20.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:20.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:20.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:20.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:20.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:20.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:19 smithi064 ceph-mon[96720]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:20.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:20.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:19:20.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:19:20.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:19:20.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:19:20.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: pgmap v79: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:20.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:19 smithi144 ceph-mon[98417]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:19:21.731 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:21 smithi064 ceph-mon[96720]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 8.1 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-07-14T18:19:21.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:21 smithi105 ceph-mon[98013]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 8.1 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-07-14T18:19:21.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:21 smithi144 ceph-mon[98417]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 8.1 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-07-14T18:19:23.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:19:23.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:23.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:19:23.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:23.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:19:23.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:24.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:24 smithi105 ceph-mon[98013]: Deploying daemon osd.3 on smithi064 2023-07-14T18:19:24.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:24 smithi105 ceph-mon[98013]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 264 KiB/s wr, 18 op/s 2023-07-14T18:19:24.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:24 smithi064 ceph-mon[96720]: Deploying daemon osd.3 on smithi064 2023-07-14T18:19:24.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:24 smithi064 ceph-mon[96720]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 264 KiB/s wr, 18 op/s 2023-07-14T18:19:24.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:24 smithi144 ceph-mon[98417]: Deploying daemon osd.3 on smithi064 2023-07-14T18:19:24.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:24 smithi144 ceph-mon[98417]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 5.8 KiB/s rd, 264 KiB/s wr, 18 op/s 2023-07-14T18:19:25.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:25 smithi105 ceph-mon[98013]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-07-14T18:19:25.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:25 smithi064 ceph-mon[96720]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-07-14T18:19:25.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:25 smithi144 ceph-mon[98417]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-07-14T18:19:28.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:28 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:28 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:28 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:28.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:28 smithi064 ceph-mon[96720]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-07-14T18:19:28.678 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:28 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:28 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:28 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:28.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:28 smithi144 ceph-mon[98417]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-07-14T18:19:28.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:28 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:28 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:28 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:28 smithi105 ceph-mon[98013]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.7 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-07-14T18:19:30.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.060+0000 7f19397fa700 1 -- 172.21.15.64:0/1940279387 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f193c102290 con 0x7f19240a1140 2023-07-14T18:19:30.110 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 3 on host 'smithi064' 2023-07-14T18:19:30.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 msgr2=0x7f19240a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:30.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 0x7f19240a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f192c005950 tx=0x7f192c0058e0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 msgr2=0x7f193c195f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:30.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f193000e9d0 tx=0x7f193000ece0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 shutdown_connections 2023-07-14T18:19:30.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f19240a1140 0x7f19240a35f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.112 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f193c105980 0x7f193c19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f193c100740 0x7f193c1959f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 --2- 172.21.15.64:0/1940279387 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f193c103060 0x7f193c195f30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 >> 172.21.15.64:0/1940279387 conn(0x7f193c0fa1e0 msgr2=0x7f193c101b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 shutdown_connections 2023-07-14T18:19:30.113 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:19:30.063+0000 7f19427e5700 1 -- 172.21.15.64:0/1940279387 wait complete. 2023-07-14T18:19:30.590 DEBUG:teuthology.orchestra.run.smithi064:osd.3> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T18:19:30.592 INFO:tasks.cephadm:Deploying osd.4 on smithi105 with /dev/vg_nvme/lv_3... 2023-07-14T18:19:30.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:19:30.624 INFO:journalctl@ceph.osd.3.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:19:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T18:19:30.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:30.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:30.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:30.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-07-14T18:19:30.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:30.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:30.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:30.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:31.157 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:19:31.588 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:31 smithi064 ceph-mon[96720]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:19:31.589 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:19:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:19:31.302+0000 7f0f88371640 -1 osd.3 0 log_to_monitors true 2023-07-14T18:19:31.634 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:31 smithi105 ceph-mon[98013]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:19:31.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:31 smithi144 ceph-mon[98417]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:19:32.667 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:19:32.667 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:19:32.667 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2023-07-14T18:19:32.667 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2023-07-14T18:19:32.667 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111126 s, 944 MB/s 2023-07-14T18:19:32.668 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2023-07-14T18:19:32.704 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:32 smithi105 ceph-mon[98013]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:19:32.704 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:32 smithi105 ceph-mon[98013]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:19:32.704 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:32 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:32.705 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:32 smithi105 ceph-mon[98013]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:19:32.705 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:32 smithi105 ceph-mon[98013]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:32.803 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:19:32.394+0000 7f0f7d9e9700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T18:19:32.804 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:19:32.399+0000 7f0f777db700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:19:32.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-mon[96720]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:19:32.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-mon[96720]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:19:32.804 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:32.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-mon[96720]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:19:32.805 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:32 smithi064 ceph-mon[96720]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:32.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:32 smithi144 ceph-mon[98417]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:19:32.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:32 smithi144 ceph-mon[98417]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:19:32.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:32 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:32.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:32 smithi144 ceph-mon[98417]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:19:32.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:32 smithi144 ceph-mon[98417]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:19:33.100 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2023-07-14T18:19:33.647 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:19:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:19:33.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:33.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:33.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:33 smithi105 ceph-mon[98013]: osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] boot 2023-07-14T18:19:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:19:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:19:33.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:33.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:33.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:33 smithi064 ceph-mon[96720]: osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] boot 2023-07-14T18:19:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:19:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:19:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:33.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:33 smithi144 ceph-mon[98417]: osd.3 [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] boot 2023-07-14T18:19:34.787 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 -- 172.21.15.105:0/725099780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe030100740 msgr2=0x7fe030107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:34.787 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 --2- 172.21.15.105:0/725099780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe030100740 0x7fe030107000 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe02800b0a0 tx=0x7fe02800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.787 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 -- 172.21.15.105:0/725099780 shutdown_connections 2023-07-14T18:19:34.787 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 --2- 172.21.15.105:0/725099780 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 0x7fe0300fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.788 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 --2- 172.21.15.105:0/725099780 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe030100740 0x7fe030107000 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.788 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 --2- 172.21.15.105:0/725099780 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0300ffdb0 0x7fe030100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.788 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 -- 172.21.15.105:0/725099780 >> 172.21.15.105:0/725099780 conn(0x7fe0300fa1e0 msgr2=0x7fe0300fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:34.788 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 -- 172.21.15.105:0/725099780 shutdown_connections 2023-07-14T18:19:34.788 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.738+0000 7fe0383d6700 1 -- 172.21.15.105:0/725099780 wait complete. 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 Processor -- start 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 -- start start 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 0x7fe030069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe030100740 0x7fe03019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:34.790 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03019d510 con 0x7fe030100740 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03019d680 con 0x7fe0300ffdb0 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0383d6700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03019d840 con 0x7fe0300febb0 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57196/0 (socket says 172.21.15.105:57196) 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 -- 172.21.15.105:0/4241141672 learned_addr learned my addr 172.21.15.105:0/4241141672 (peer_addr_for_me v2:172.21.15.105:0/0) 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe036973700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe030100740 0x7fe03019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:34.791 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe036172700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 0x7fe030069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 -- 172.21.15.105:0/4241141672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 msgr2=0x7fe030069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 0x7fe030069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 -- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe030100740 msgr2=0x7fe03019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe030100740 0x7fe03019cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 -- 172.21.15.105:0/4241141672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009d00 con 0x7fe0300ffdb0 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe035971700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00e9a0 tx=0x7fe02c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:34.792 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.739+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c009790 con 0x7fe0300ffdb0 2023-07-14T18:19:34.793 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.740+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03019db20 con 0x7fe0300ffdb0 2023-07-14T18:19:34.793 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.740+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe02c010430 con 0x7fe0300ffdb0 2023-07-14T18:19:34.793 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.740+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02c00cea0 con 0x7fe0300ffdb0 2023-07-14T18:19:34.793 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.740+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03019e020 con 0x7fe0300ffdb0 2023-07-14T18:19:34.793 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.740+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03010a1e0 con 0x7fe0300ffdb0 2023-07-14T18:19:34.794 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.741+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe02c0105a0 con 0x7fe0300ffdb0 2023-07-14T18:19:34.794 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.742+0000 7fe0277fe700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 0x7fe01c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:34.794 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.742+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fe02c014070 con 0x7fe0300ffdb0 2023-07-14T18:19:34.795 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.742+0000 7fe036172700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 0x7fe01c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:34.795 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.743+0000 7fe036172700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 0x7fe01c0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0300ffc10 tx=0x7fe02000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:34.795 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.746+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe02c096290 con 0x7fe0300ffdb0 2023-07-14T18:19:34.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:19:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:19:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi064 2023-07-14T18:19:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi064 to 8584M 2023-07-14T18:19:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:19:34.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:34.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:34.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:34 smithi105 ceph-mon[98013]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:19:34.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:19:34.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:19:34.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi064 2023-07-14T18:19:34.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi064 to 8584M 2023-07-14T18:19:34.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:19:34.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:34.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:34.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:34 smithi064 ceph-mon[96720]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:19:34.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:19:34.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:19:34.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi064 2023-07-14T18:19:34.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi064 to 8584M 2023-07-14T18:19:34.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:19:34.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:19:34.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:34.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:34 smithi144 ceph-mon[98417]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:19:34.990 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:34.941+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe030060a80 con 0x7fe01c0a1150 2023-07-14T18:19:36.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:19:36.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:36.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:36.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:36.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:36.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:35 smithi105 ceph-mon[98013]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:36.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:19:36.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:36.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:36.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:36.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:36.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:35 smithi064 ceph-mon[96720]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:36.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:19:36.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:36.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:36.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:36.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:36.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:35 smithi144 ceph-mon[98417]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:37.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:36 smithi105 ceph-mon[98013]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:19:37.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:36 smithi064 ceph-mon[96720]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:19:37.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:36 smithi144 ceph-mon[98417]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:19:38.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:19:38.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/542109378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]': finished 2023-07-14T18:19:38.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:19:38.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:38.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2023-07-14T18:19:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:37 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/1078795408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:38.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:19:38.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/542109378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]': finished 2023-07-14T18:19:38.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:19:38.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:38.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2023-07-14T18:19:38.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:37 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/1078795408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:38.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:19:38.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/542109378' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]: dispatch 2023-07-14T18:19:38.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f57e7434-c154-40a8-8b1e-13901e8bc9e7"}]': finished 2023-07-14T18:19:38.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:19:38.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:38.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 166 KiB/s, 0 objects/s recovering 2023-07-14T18:19:38.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:37 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/1078795408' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:19:40.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:39 smithi105 ceph-mon[98013]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:19:40.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:39 smithi064 ceph-mon[96720]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:19:40.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:39 smithi144 ceph-mon[98417]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-07-14T18:19:41.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:41 smithi105 ceph-mon[98013]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:19:41.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:41 smithi064 ceph-mon[96720]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:19:41.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:41 smithi144 ceph-mon[98417]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:19:43.608 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:43 smithi105 ceph-mon[98013]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:19:43.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:43 smithi064 ceph-mon[96720]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:19:43.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:43 smithi144 ceph-mon[98417]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:19:44.557 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:19:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:44.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:44 smithi105 ceph-mon[98013]: Deploying daemon osd.4 on smithi105 2023-07-14T18:19:44.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:19:44.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:44.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:44 smithi064 ceph-mon[96720]: Deploying daemon osd.4 on smithi105 2023-07-14T18:19:44.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:19:44.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:44.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:44 smithi144 ceph-mon[98417]: Deploying daemon osd.4 on smithi105 2023-07-14T18:19:45.793 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:45 smithi105 ceph-mon[98013]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:45.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:45 smithi064 ceph-mon[96720]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:45.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:45 smithi144 ceph-mon[98417]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:47.778 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:47 smithi105 ceph-mon[98013]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:47.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:47 smithi064 ceph-mon[96720]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:47.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:47 smithi144 ceph-mon[98417]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:49.041 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:48 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.041 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:48 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.041 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:48 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:49.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:49.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:49.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:50.057 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:49 smithi105 ceph-mon[98013]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:50.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.058 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:49 smithi064 ceph-mon[96720]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:50.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:49 smithi144 ceph-mon[98417]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:50.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:50.767 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 4 on host 'smithi105' 2023-07-14T18:19:50.767 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.717+0000 7fe0277fe700 1 -- 172.21.15.105:0/4241141672 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe030060a80 con 0x7fe01c0a1150 2023-07-14T18:19:50.768 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.719+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 msgr2=0x7fe01c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:50.768 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.719+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 0x7fe01c0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe0300ffc10 tx=0x7fe02000a380 comp rx=0 tx=0).stop 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.719+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 msgr2=0x7fe03019a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.719+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe02c00e9a0 tx=0x7fe02c00ecb0 comp rx=0 tx=0).stop 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 shutdown_connections 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe0300febb0 0x7fe030069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe01c0a1150 0x7fe01c0a3600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe030100740 0x7fe03019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:50.769 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 --2- 172.21.15.105:0/4241141672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0300ffdb0 0x7fe03019a6e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:50.770 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 >> 172.21.15.105:0/4241141672 conn(0x7fe0300fa1e0 msgr2=0x7fe0300fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:50.770 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 shutdown_connections 2023-07-14T18:19:50.770 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:19:50.720+0000 7fe0383d6700 1 -- 172.21.15.105:0/4241141672 wait complete. 2023-07-14T18:19:51.230 DEBUG:teuthology.orchestra.run.smithi105:osd.4> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T18:19:51.233 INFO:tasks.cephadm:Deploying osd.5 on smithi144 with /dev/vg_nvme/lv_3... 2023-07-14T18:19:51.234 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:19:51.257 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:51.257 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:51.257 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.257 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.258 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.258 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:50 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:51.262 INFO:journalctl@ceph.osd.4.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:19:51.266 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:51.266 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:51.266 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.267 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.267 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.267 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:50 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:51.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:51.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:51.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:51.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:50 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:19:51.793 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:19:51.957 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:19:51 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:19:51.763+0000 7f84f034b640 -1 osd.4 0 log_to_monitors true 2023-07-14T18:19:52.229 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:51 smithi105 ceph-mon[98013]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:52.230 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:51 smithi105 ceph-mon[98013]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:52.230 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:51 smithi105 ceph-mon[98013]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:52.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:51 smithi064 ceph-mon[96720]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:52.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:51 smithi064 ceph-mon[96720]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:52.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:51 smithi064 ceph-mon[96720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:52.350 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:51 smithi144 ceph-mon[98417]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:52.351 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:51 smithi144 ceph-mon[98417]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:52.351 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:51 smithi144 ceph-mon[98417]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:19:53.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:52 smithi144 ceph-mon[98417]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:19:53.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:52 smithi144 ceph-mon[98417]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:19:53.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:52 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:53.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:52 smithi144 ceph-mon[98417]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:52 smithi144 ceph-mon[98417]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:19:52.962+0000 7f84e59c3700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T18:19:53.308 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:19:52.968+0000 7f84dd7b1700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:19:53.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-mon[98013]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:19:53.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-mon[98013]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:19:53.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:53.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-mon[98013]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:52 smithi105 ceph-mon[98013]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:52 smithi064 ceph-mon[96720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:19:53.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:52 smithi064 ceph-mon[96720]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:19:53.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:52 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:53.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:52 smithi064 ceph-mon[96720]: from='osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:52 smithi064 ceph-mon[96720]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:19:53.400 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:19:53.401 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:19:53.401 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2023-07-14T18:19:53.401 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2023-07-14T18:19:53.402 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110968 s, 945 MB/s 2023-07-14T18:19:53.402 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2023-07-14T18:19:53.835 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi144:vg_nvme/lv_3 2023-07-14T18:19:54.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:19:54.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:19:54.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:54.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:54.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:19:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:19:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:54.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:54.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:19:54.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:54.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:19:54.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:19:54.439 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:19:55.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:19:55.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:19:55.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi105 2023-07-14T18:19:55.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi105 to 8584M 2023-07-14T18:19:55.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] boot 2023-07-14T18:19:55.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:19:55.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:54 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:55.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi105 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi105 to 8584M 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] boot 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:19:55.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:54 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:55.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi105 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi105 to 8584M 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: osd.4 [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] boot 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:19:55.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:54 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:19:55.795 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 -- 172.21.15.144:0/1169823139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c0fee90 msgr2=0x7f2c3c0ff2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:55.795 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 --2- 172.21.15.144:0/1169823139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c0ff2e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2c300099c0 tx=0x7f2c30009cd0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.795 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 -- 172.21.15.144:0/1169823139 shutdown_connections 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 --2- 172.21.15.144:0/1169823139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 0x7f2c3c1070f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 --2- 172.21.15.144:0/1169823139 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c100150 0x7f2c3c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 --2- 172.21.15.144:0/1169823139 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c0ff2e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 -- 172.21.15.144:0/1169823139 >> 172.21.15.144:0/1169823139 conn(0x7f2c3c0f9990 msgr2=0x7f2c3c0fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 -- 172.21.15.144:0/1169823139 shutdown_connections 2023-07-14T18:19:55.796 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.747+0000 7f2c425f8700 1 -- 172.21.15.144:0/1169823139 wait complete. 2023-07-14T18:19:55.798 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 Processor -- start 2023-07-14T18:19:55.798 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 -- start start 2023-07-14T18:19:55.798 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 0x7f2c3c194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c19c3c0 con 0x7f2c3c0ff820 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c19c500 con 0x7f2c3c0fee90 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c425f8700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c3c19c690 con 0x7f2c3c100150 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c40b95700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c40b95700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54820/0 (socket says 172.21.15.144:54820) 2023-07-14T18:19:55.799 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c40b95700 1 -- 172.21.15.144:0/4282596755 learned_addr learned my addr 172.21.15.144:0/4282596755 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.748+0000 7f2c3bfff700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c3b7fe700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 0x7f2c3c194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 -- 172.21.15.144:0/4282596755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c0fee90 msgr2=0x7f2c3c194830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c194830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 -- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 msgr2=0x7f2c3c194d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 0x7f2c3c194d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:19:55.800 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 -- 172.21.15.144:0/4282596755 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c300095f0 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c40b95700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2c34007f10 tx=0x7f2c3400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34017070 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2c3400f040 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c3c19c970 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c34005570 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.749+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c3c19ce70 con 0x7f2c3c100150 2023-07-14T18:19:55.801 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.750+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c3c1062f0 con 0x7f2c3c100150 2023-07-14T18:19:55.802 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.751+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2c3400db20 con 0x7f2c3c100150 2023-07-14T18:19:55.802 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.752+0000 7f2c397fa700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 0x7f2c240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:19:55.802 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.752+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f2c340caff0 con 0x7f2c3c100150 2023-07-14T18:19:55.803 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.752+0000 7f2c3bfff700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 0x7f2c240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:19:55.803 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.752+0000 7f2c3bfff700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 0x7f2c240a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c067e80 tx=0x7f2c2c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:19:55.803 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.756+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2c340957b0 con 0x7f2c3c100150 2023-07-14T18:19:55.998 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:19:55.951+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c3c060a80 con 0x7f2c240a1150 2023-07-14T18:19:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:55 smithi105 ceph-mon[98013]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:19:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:55 smithi105 ceph-mon[98013]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:56.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:56.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:55 smithi064 ceph-mon[96720]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:19:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:55 smithi064 ceph-mon[96720]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:56.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:55 smithi144 ceph-mon[98417]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:19:56.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:55 smithi144 ceph-mon[98417]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:56.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:19:56.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:19:56.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:19:57.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:56 smithi144 ceph-mon[98417]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:57.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:56 smithi105 ceph-mon[98013]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:57.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:56 smithi064 ceph-mon[96720]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:19:58.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:57 smithi105 ceph-mon[98013]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:58.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:57 smithi064 ceph-mon[96720]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:58.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:57 smithi144 ceph-mon[98417]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:19:59.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:58 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/2224822133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:58 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:58 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]': finished 2023-07-14T18:19:59.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:58 smithi105 ceph-mon[98013]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:19:59.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:19:59.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:58 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/2224822133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:58 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:58 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]': finished 2023-07-14T18:19:59.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:58 smithi064 ceph-mon[96720]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:19:59.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:19:59.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:58 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/2224822133' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:58 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]: dispatch 2023-07-14T18:19:59.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:58 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb"}]': finished 2023-07-14T18:19:59.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:58 smithi144 ceph-mon[98417]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:19:59.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:59 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/506453717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:00.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:19:59 smithi105 ceph-mon[98013]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:00.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:59 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/506453717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:00.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:19:59 smithi064 ceph-mon[96720]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:00.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:59 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/506453717' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:00.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:19:59 smithi144 ceph-mon[98417]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:01.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:00 smithi105 ceph-mon[98013]: overall HEALTH_OK 2023-07-14T18:20:01.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:00 smithi064 ceph-mon[96720]: overall HEALTH_OK 2023-07-14T18:20:01.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:00 smithi144 ceph-mon[98417]: overall HEALTH_OK 2023-07-14T18:20:02.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:02 smithi105 ceph-mon[98013]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:02.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:01 smithi064 ceph-mon[96720]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:02.365 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:02 smithi144 ceph-mon[98417]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:03.756 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:03 smithi144 ceph-mon[98417]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:03.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:03 smithi105 ceph-mon[98013]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:03.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:03 smithi064 ceph-mon[96720]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:05.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:05 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:20:05.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:05 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:05.721 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:05 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:20:05.721 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:05 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:05.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:20:05.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:05 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:06.555 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:06 smithi144 ceph-mon[98417]: Deploying daemon osd.5 on smithi144 2023-07-14T18:20:06.556 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:06 smithi144 ceph-mon[98417]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:06.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:06 smithi105 ceph-mon[98013]: Deploying daemon osd.5 on smithi144 2023-07-14T18:20:06.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:06 smithi105 ceph-mon[98013]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:06.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:06 smithi064 ceph-mon[96720]: Deploying daemon osd.5 on smithi144 2023-07-14T18:20:06.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:06 smithi064 ceph-mon[96720]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:07.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:07 smithi105 ceph-mon[98013]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:07.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:07 smithi064 ceph-mon[96720]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:07.844 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:07 smithi144 ceph-mon[98417]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:10.004 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:09 smithi144 ceph-mon[98417]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:10.307 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:09 smithi105 ceph-mon[98013]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:10.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:09 smithi064 ceph-mon[96720]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:11.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:11.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:11.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:11.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:12.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:12 smithi064 ceph-mon[96720]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:12.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:12 smithi144 ceph-mon[98417]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:12.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:12 smithi105 ceph-mon[98013]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:12.963 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.914+0000 7f2c397fa700 1 -- 172.21.15.144:0/4282596755 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2c3c060a80 con 0x7f2c240a1150 2023-07-14T18:20:12.964 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 5 on host 'smithi144' 2023-07-14T18:20:12.966 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 msgr2=0x7f2c240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:12.966 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 0x7f2c240a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c067e80 tx=0x7f2c2c00b410 comp rx=0 tx=0).stop 2023-07-14T18:20:12.966 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 msgr2=0x7f2c3c19bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:12.966 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2c34007f10 tx=0x7f2c3400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:20:12.966 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 shutdown_connections 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2c3c100150 0x7f2c3c19bdf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2c240a1150 0x7f2c240a3600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2c3c0fee90 0x7f2c3c194830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 --2- 172.21.15.144:0/4282596755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c3c0ff820 0x7f2c3c194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 >> 172.21.15.144:0/4282596755 conn(0x7f2c3c0f9990 msgr2=0x7f2c3c103430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 shutdown_connections 2023-07-14T18:20:12.967 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:12.917+0000 7f2c425f8700 1 -- 172.21.15.144:0/4282596755 wait complete. 2023-07-14T18:20:13.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:13.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:13.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.435 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.435 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:13.435 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.436 DEBUG:teuthology.orchestra.run.smithi144:osd.5> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T18:20:13.438 INFO:tasks.cephadm:Deploying osd.6 on smithi064 with /dev/vg_nvme/lv_2... 2023-07-14T18:20:13.439 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:20:13.469 INFO:journalctl@ceph.osd.5.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:20:13.485 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:13.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:13.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.486 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:13.487 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:13.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:13.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:13.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:14.032 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:20:14.179 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:20:14 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:20:13.999+0000 7fe205bdd640 -1 osd.5 0 log_to_monitors true 2023-07-14T18:20:14.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:14 smithi064 ceph-mon[96720]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:14.447 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:14 smithi064 ceph-mon[96720]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:14.448 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:14 smithi064 ceph-mon[96720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:14.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:14 smithi105 ceph-mon[98013]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:14.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:14 smithi105 ceph-mon[98013]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:14.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:14 smithi105 ceph-mon[98013]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:14.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:14 smithi144 ceph-mon[98417]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:14.679 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:14 smithi144 ceph-mon[98417]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:14.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:14 smithi144 ceph-mon[98417]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111762 s, 938 MB/s 2023-07-14T18:20:15.774 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2023-07-14T18:20:16.088 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:20:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:20:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:16.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:15 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2023-07-14T18:20:16.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:15 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.430 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:20:15.955+0000 7fe1fb255700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T18:20:16.430 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:20:15.961+0000 7fe1f704b700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:20:16.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:16.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:16.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:15 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:20:17.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi144 2023-07-14T18:20:17.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi144 to 8584M 2023-07-14T18:20:17.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:20:17.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:20:17.283 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:17.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:17.284 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:16 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi144 2023-07-14T18:20:17.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi144 to 8584M 2023-07-14T18:20:17.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:20:17.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:17.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:16 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi144 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi144 to 8584M 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:17.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:17.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:17.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:16 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:17.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 -- 172.21.15.64:0/3774348849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 msgr2=0x7f1d4c107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:17.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3774348849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c107de0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d4400b0a0 tx=0x7f1d4400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.957 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 -- 172.21.15.64:0/3774348849 shutdown_connections 2023-07-14T18:20:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3774348849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c107de0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3774348849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1007a0 0x7f1d4c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3774348849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1030c0 0x7f1d4c1054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.908+0000 7f1d522cb700 1 -- 172.21.15.64:0/3774348849 >> 172.21.15.64:0/3774348849 conn(0x7f1d4c0fa1e0 msgr2=0x7f1d4c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:17.958 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 -- 172.21.15.64:0/3774348849 shutdown_connections 2023-07-14T18:20:17.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 -- 172.21.15.64:0/3774348849 wait complete. 2023-07-14T18:20:17.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 Processor -- start 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 -- start start 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1007a0 0x7f1d4c1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1030c0 0x7f1d4c195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.909+0000 7f1d522cb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d522cb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c19d530 con 0x7f1d4c1059e0 2023-07-14T18:20:17.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d522cb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c19d670 con 0x7f1d4c1030c0 2023-07-14T18:20:17.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d522cb700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d4c19d800 con 0x7f1d4c1007a0 2023-07-14T18:20:17.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:17.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46148/0 (socket says 172.21.15.64:46148) 2023-07-14T18:20:17.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d4b7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1030c0 0x7f1d4c195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:17.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 -- 172.21.15.64:0/3465796318 learned_addr learned my addr 172.21.15.64:0/3465796318 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d4bfff700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1007a0 0x7f1d4c1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 -- 172.21.15.64:0/3465796318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1007a0 msgr2=0x7f1d4c1959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1007a0 0x7f1d4c1959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 -- 172.21.15.64:0/3465796318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1030c0 msgr2=0x7f1d4c195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1030c0 0x7f1d4c195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 -- 172.21.15.64:0/3465796318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d44009d00 con 0x7f1d4c1059e0 2023-07-14T18:20:17.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d50868700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006070 tx=0x7f1d44004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4400e040 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1d44003e90 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d4c19da80 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4401de90 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.910+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d4c19df40 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.911+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4c0fbd50 con 0x7f1d4c1059e0 2023-07-14T18:20:17.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.913+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1d4401d690 con 0x7f1d4c1059e0 2023-07-14T18:20:17.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.913+0000 7f1d497fa700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 0x7f1d340a7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:17.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.913+0000 7f1d4bfff700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 0x7f1d340a7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:17.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.914+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4566+0+0 (secure 0 0 0) 0x7f1d440cd1d0 con 0x7f1d4c1059e0 2023-07-14T18:20:17.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.914+0000 7f1d4bfff700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 0x7f1d340a7a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c005950 tx=0x7f1d3c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:17.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:17.918+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1d44097850 con 0x7f1d4c1059e0 2023-07-14T18:20:18.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:18.127+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d4c060a80 con 0x7f1d340a55e0 2023-07-14T18:20:18.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:20:18.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:20:18.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] boot 2023-07-14T18:20:18.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:20:18.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:17 smithi105 ceph-mon[98013]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] boot 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:20:18.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:17 smithi064 ceph-mon[96720]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:18.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:20:18.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:20:18.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: osd.5 [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] boot 2023-07-14T18:20:18.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:20:18.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:20:18.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:17 smithi144 ceph-mon[98417]: pgmap v124: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:20:19.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:18 smithi105 ceph-mon[98013]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:20:19.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:18 smithi105 ceph-mon[98013]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:19.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:18 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:19.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:18 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:19.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:18 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:18 smithi064 ceph-mon[96720]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:20:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:18 smithi064 ceph-mon[96720]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:19.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:19.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:19.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:18 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:19.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:18 smithi144 ceph-mon[98417]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:20:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:18 smithi144 ceph-mon[98417]: from='client.14364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:19.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:18 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:19 smithi105 ceph-mon[98013]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:20:20.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:19 smithi105 ceph-mon[98013]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:20.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:19 smithi064 ceph-mon[96720]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:20:20.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:19 smithi064 ceph-mon[96720]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:20.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:19 smithi144 ceph-mon[98417]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:20:20.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:19 smithi144 ceph-mon[98417]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:21.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:20 smithi105 ceph-mon[98013]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:20:21.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:20 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]: dispatch 2023-07-14T18:20:21.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:20 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]': finished 2023-07-14T18:20:21.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:20 smithi105 ceph-mon[98013]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:20:21.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:21.338 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:20 smithi064 ceph-mon[96720]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:20:21.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]: dispatch 2023-07-14T18:20:21.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]': finished 2023-07-14T18:20:21.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:20 smithi064 ceph-mon[96720]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:20:21.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:21.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:20 smithi144 ceph-mon[98417]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:20:21.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:20 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]: dispatch 2023-07-14T18:20:21.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:20 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/18028979' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "13faa2e5-0f41-43f8-a486-b5404b338724"}]': finished 2023-07-14T18:20:21.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:20 smithi144 ceph-mon[98417]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:20:21.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:22.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:21 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2013451509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:22.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:21 smithi105 ceph-mon[98013]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:20:22.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:21 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2013451509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:22.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:21 smithi064 ceph-mon[96720]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:20:22.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:21 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2013451509' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:22.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:21 smithi144 ceph-mon[98417]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-07-14T18:20:23.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:23 smithi105 ceph-mon[98013]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-14T18:20:23.838 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:23 smithi064 ceph-mon[96720]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-14T18:20:23.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:23 smithi144 ceph-mon[98417]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-07-14T18:20:25.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:25 smithi105 ceph-mon[98013]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:20:25.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:25 smithi064 ceph-mon[96720]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:20:25.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:25 smithi144 ceph-mon[98417]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T18:20:27.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:27 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:20:27.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:27 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:27.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:27 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:20:27.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:27 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:27.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:27 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:20:27.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:27 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:28 smithi105 ceph-mon[98013]: Deploying daemon osd.6 on smithi064 2023-07-14T18:20:28.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:28 smithi105 ceph-mon[98013]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:20:28.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:28 smithi064 ceph-mon[96720]: Deploying daemon osd.6 on smithi064 2023-07-14T18:20:28.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:28 smithi064 ceph-mon[96720]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:20:28.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:28 smithi144 ceph-mon[98417]: Deploying daemon osd.6 on smithi064 2023-07-14T18:20:28.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:28 smithi144 ceph-mon[98417]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:20:30.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:29 smithi105 ceph-mon[98013]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:20:30.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:29 smithi064 ceph-mon[96720]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:20:30.429 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:29 smithi144 ceph-mon[98417]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-07-14T18:20:31.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:31 smithi105 ceph-mon[98013]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T18:20:31.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:31 smithi064 ceph-mon[96720]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T18:20:31.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:31 smithi144 ceph-mon[98417]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T18:20:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.064 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:32 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:33.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:32 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:32 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:32 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:33.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:32 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:32 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:33.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:32 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:34.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:33 smithi064 ceph-mon[96720]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:34.179 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:33 smithi144 ceph-mon[98417]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:34.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:33 smithi105 ceph-mon[98013]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:34.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.739+0000 7f1d497fa700 1 -- 172.21.15.64:0/3465796318 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1d4c060a80 con 0x7f1d340a55e0 2023-07-14T18:20:34.788 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 6 on host 'smithi064' 2023-07-14T18:20:34.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 msgr2=0x7f1d340a7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:34.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 0x7f1d340a7a90 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c005950 tx=0x7f1d3c0058e0 comp rx=0 tx=0).stop 2023-07-14T18:20:34.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 msgr2=0x7f1d4c19cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:34.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d44006070 tx=0x7f1d44004d00 comp rx=0 tx=0).stop 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 shutdown_connections 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1d4c1007a0 0x7f1d4c1959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1d340a55e0 0x7f1d340a7a90 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d4c1059e0 0x7f1d4c19cf60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 --2- 172.21.15.64:0/3465796318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1d4c1030c0 0x7f1d4c195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 >> 172.21.15.64:0/3465796318 conn(0x7f1d4c0fa1e0 msgr2=0x7f1d4c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:34.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 shutdown_connections 2023-07-14T18:20:34.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:20:34.741+0000 7f1d522cb700 1 -- 172.21.15.64:0/3465796318 wait complete. 2023-07-14T18:20:35.251 DEBUG:teuthology.orchestra.run.smithi064:osd.6> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T18:20:35.254 INFO:tasks.cephadm:Deploying osd.7 on smithi105 with /dev/vg_nvme/lv_2... 2023-07-14T18:20:35.254 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:20:35.284 INFO:journalctl@ceph.osd.6.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:20:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:35.834 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:20:36.089 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:20:35 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:20:35.840+0000 7fa1aaa82640 -1 osd.6 0 log_to_monitors true 2023-07-14T18:20:36.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:36 smithi064 ceph-mon[96720]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:36.433 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:36 smithi064 ceph-mon[96720]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:20:36.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:36 smithi105 ceph-mon[98013]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:36.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:36 smithi105 ceph-mon[98013]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:20:36.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:36 smithi144 ceph-mon[98417]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:36.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:36 smithi144 ceph-mon[98417]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T18:20:37.328 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:20:37.328 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:20:37.329 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2023-07-14T18:20:37.329 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2023-07-14T18:20:37.329 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143572 s, 730 MB/s 2023-07-14T18:20:37.329 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2023-07-14T18:20:37.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-mon[96720]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:20:37.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-mon[96720]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:20:37.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:37.549 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-mon[96720]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:20:37.550 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:20:37.390+0000 7fa1a00fa700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T18:20:37.550 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:20:37 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:20:37.398+0000 7fa1986e9700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:20:37.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:37 smithi105 ceph-mon[98013]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:20:37.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:37 smithi105 ceph-mon[98013]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:20:37.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:37.668 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:37 smithi105 ceph-mon[98013]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:20:37.685 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2023-07-14T18:20:37.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:37 smithi144 ceph-mon[98417]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T18:20:37.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:37 smithi144 ceph-mon[98417]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T18:20:37.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:37.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:37 smithi144 ceph-mon[98417]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:20:38.239 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:20:38.743 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:20:38.743 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.744 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.745 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.745 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.745 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.745 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.746 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.746 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:38.746 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.746 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.746 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:20:38.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:38.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:20:38.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T18:20:38.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:38.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:38.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 -- 172.21.15.105:0/2432024765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb8100740 msgr2=0x7f5cb8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:39.434 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/2432024765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb8100740 0x7f5cb8107000 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5cb000b0a0 tx=0x7f5cb000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 -- 172.21.15.105:0/2432024765 shutdown_connections 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/2432024765 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb80febb0 0x7f5cb80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/2432024765 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb8100740 0x7f5cb8107000 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/2432024765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb8100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 -- 172.21.15.105:0/2432024765 >> 172.21.15.105:0/2432024765 conn(0x7f5cb80fa1e0 msgr2=0x7f5cb80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 -- 172.21.15.105:0/2432024765 shutdown_connections 2023-07-14T18:20:39.435 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 -- 172.21.15.105:0/2432024765 wait complete. 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.384+0000 7f5cbfd22700 1 Processor -- start 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 -- start start 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb8100740 0x7f5cb819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb819d530 con 0x7f5cb80ffdb0 2023-07-14T18:20:39.436 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb819d6a0 con 0x7f5cb80febb0 2023-07-14T18:20:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb819d860 con 0x7f5cb8100740 2023-07-14T18:20:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57292/0 (socket says 172.21.15.105:57292) 2023-07-14T18:20:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 -- 172.21.15.105:0/3506656815 learned_addr learned my addr 172.21.15.105:0/3506656815 (peer_addr_for_me v2:172.21.15.105:0/0) 2023-07-14T18:20:39.437 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbe2bf700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb8100740 0x7f5cb819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:39.438 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbd2bd700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 -- 172.21.15.105:0/3506656815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb8100740 msgr2=0x7f5cb819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb8100740 0x7f5cb819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 -- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 msgr2=0x7f5cb819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 -- 172.21.15.105:0/3506656815 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb0009d00 con 0x7f5cb80febb0 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbd2bd700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:20:39.439 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbdabe700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800d980 tx=0x7f5ca800dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8004020 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ca8009730 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb819db40 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.385+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca8003a00 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.386+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb819e040 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.386+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb81061d0 con 0x7f5cb80febb0 2023-07-14T18:20:39.440 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.388+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5ca80098a0 con 0x7f5cb80febb0 2023-07-14T18:20:39.441 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.388+0000 7f5caeffd700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 0x7f5ca40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:39.441 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.388+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f5ca80cbd00 con 0x7f5cb80febb0 2023-07-14T18:20:39.441 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.388+0000 7f5cbd2bd700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 0x7f5ca40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:39.441 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.389+0000 7f5cbd2bd700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 0x7f5ca40a3600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005fd0 tx=0x7f5cb4005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:39.441 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.391+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5ca80960c0 con 0x7f5cb80febb0 2023-07-14T18:20:39.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:20:39.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:20:39.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi064 2023-07-14T18:20:39.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi064 to 5722M 2023-07-14T18:20:39.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] boot 2023-07-14T18:20:39.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:20:39.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:39.637 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:39.588+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5cb8060a80 con 0x7f5ca40a1150 2023-07-14T18:20:39.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:20:39.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:20:39.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi064 2023-07-14T18:20:39.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi064 to 5722M 2023-07-14T18:20:39.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] boot 2023-07-14T18:20:39.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:20:39.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:39.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi064 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi064 to 5722M 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: osd.6 [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] boot 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T18:20:39.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:20:40.703 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:40 smithi105 ceph-mon[98013]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:20:40.703 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:40 smithi105 ceph-mon[98013]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:40.703 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:40.703 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:40.703 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:40.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:40 smithi064 ceph-mon[96720]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:20:40.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:40 smithi064 ceph-mon[96720]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:40.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:40.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:40.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:40.929 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:40 smithi144 ceph-mon[98417]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T18:20:40.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:40 smithi144 ceph-mon[98417]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:40.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:20:40.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:20:40.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:41.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:41 smithi105 ceph-mon[98013]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:41.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:41 smithi064 ceph-mon[96720]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:41.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:41 smithi144 ceph-mon[98417]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:20:42.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:42.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/1172174669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]': finished 2023-07-14T18:20:42.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:20:42.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:42.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:42.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/1172174669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]': finished 2023-07-14T18:20:42.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:20:42.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:42.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:42.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/1172174669' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]: dispatch 2023-07-14T18:20:42.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3"}]': finished 2023-07-14T18:20:42.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T18:20:42.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:43.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:43 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/2552273341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:43.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:43 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/2552273341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:43.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:43 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/2552273341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:20:44.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:44 smithi105 ceph-mon[98013]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:44.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:44 smithi064 ceph-mon[96720]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:44.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:44 smithi144 ceph-mon[98417]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:46.747 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:46 smithi105 ceph-mon[98013]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:46.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:46 smithi064 ceph-mon[96720]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:46.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:46 smithi144 ceph-mon[98417]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:48.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:48 smithi105 ceph-mon[98013]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:48 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:20:48.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:48 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:48 smithi064 ceph-mon[96720]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:48.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:20:48.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:48 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:48.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:48 smithi144 ceph-mon[98417]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:48.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T18:20:48.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:48 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:49.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:49 smithi105 ceph-mon[98013]: Deploying daemon osd.7 on smithi105 2023-07-14T18:20:49.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:49 smithi064 ceph-mon[96720]: Deploying daemon osd.7 on smithi105 2023-07-14T18:20:49.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:49 smithi144 ceph-mon[98417]: Deploying daemon osd.7 on smithi105 2023-07-14T18:20:50.601 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:50 smithi105 ceph-mon[98013]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:50.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:50 smithi064 ceph-mon[96720]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:50.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:50 smithi144 ceph-mon[98417]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:51.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:51 smithi105 ceph-mon[98013]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:51.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:51 smithi064 ceph-mon[96720]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:51.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:51 smithi144 ceph-mon[98417]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:53.633 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.634 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.634 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:53.634 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:53 smithi105 ceph-mon[98013]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:53.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:53.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:53 smithi064 ceph-mon[96720]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:53.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:53.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:53.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:53 smithi144 ceph-mon[98417]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:55.156 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.107+0000 7f5caeffd700 1 -- 172.21.15.105:0/3506656815 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5cb8060a80 con 0x7f5ca40a1150 2023-07-14T18:20:55.157 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 7 on host 'smithi105' 2023-07-14T18:20:55.158 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 msgr2=0x7f5ca40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 0x7f5ca40a3600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4005fd0 tx=0x7f5cb4005dc0 comp rx=0 tx=0).stop 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 msgr2=0x7f5cb8068c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ca800d980 tx=0x7f5ca800dc90 comp rx=0 tx=0).stop 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 shutdown_connections 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5cb8100740 0x7f5cb819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ca40a1150 0x7f5ca40a3600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:55.159 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5cb80febb0 0x7f5cb8068c60 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:55.160 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 --2- 172.21.15.105:0/3506656815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5cb80ffdb0 0x7f5cb819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:55.160 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 >> 172.21.15.105:0/3506656815 conn(0x7f5cb80fa1e0 msgr2=0x7f5cb8103310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:55.160 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 shutdown_connections 2023-07-14T18:20:55.160 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:20:55.109+0000 7f5cbfd22700 1 -- 172.21.15.105:0/3506656815 wait complete. 2023-07-14T18:20:55.519 DEBUG:teuthology.orchestra.run.smithi105:osd.7> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T18:20:55.521 INFO:tasks.cephadm:Deploying osd.8 on smithi144 with /dev/vg_nvme/lv_2... 2023-07-14T18:20:55.522 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:20:55.550 INFO:journalctl@ceph.osd.7.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:20:56.089 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:20:56.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:55 smithi144 ceph-mon[98417]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:56.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.222 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:55 smithi105 ceph-mon[98013]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:56.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:55 smithi064 ceph-mon[96720]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:56.526 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:20:56 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:20:56.386+0000 7ff34aeda640 -1 osd.7 0 log_to_monitors true 2023-07-14T18:20:57.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:57 smithi144 ceph-mon[98417]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:57.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:57 smithi144 ceph-mon[98417]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:57.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:57 smithi105 ceph-mon[98013]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:57.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:57 smithi105 ceph-mon[98013]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:57.587 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:20:57.587 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:20:57.588 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2023-07-14T18:20:57.588 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2023-07-14T18:20:57.588 INFO:teuthology.orchestra.run.smithi144.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110481 s, 949 MB/s 2023-07-14T18:20:57.588 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 2023-07-14T18:20:57.588 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2023-07-14T18:20:57.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:57 smithi064 ceph-mon[96720]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:57.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:57 smithi064 ceph-mon[96720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T18:20:58.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi144:vg_nvme/lv_2 2023-07-14T18:20:58.165 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:20:58.108+0000 7ff340552700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T18:20:58.166 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:20:58.116+0000 7ff33c348700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:20:58.166 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:20:58.166 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:20:58.166 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.167 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.168 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.168 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:58.168 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:58.168 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:20:58.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:20:58.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:58.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:58.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:58.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T18:20:58.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T18:20:58.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:58.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:20:58.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T18:20:58.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:58.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:58.603 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:20:59.402 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi105 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi105 to 5722M 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:59.403 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:59.404 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:59.404 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:59.404 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] boot 2023-07-14T18:20:59.404 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:20:59.404 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:20:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi105 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi105 to 5722M 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:59.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] boot 2023-07-14T18:20:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:20:59.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:20:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi105 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi105 to 5722M 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:20:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:20:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:20:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:20:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: osd.7 [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] boot 2023-07-14T18:20:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T18:20:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:20:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T18:20:59.889 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 -- 172.21.15.144:0/817770072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c105980 msgr2=0x7fe34c107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:59.889 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 --2- 172.21.15.144:0/817770072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c105980 0x7fe34c107de0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe34400b0a0 tx=0x7fe34400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.889 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 -- 172.21.15.144:0/817770072 shutdown_connections 2023-07-14T18:20:59.889 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 --2- 172.21.15.144:0/817770072 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c105980 0x7fe34c107de0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.890 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 --2- 172.21.15.144:0/817770072 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 0x7fe34c102b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.890 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 --2- 172.21.15.144:0/817770072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c103060 0x7fe34c105440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.890 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 -- 172.21.15.144:0/817770072 >> 172.21.15.144:0/817770072 conn(0x7fe34c0fa1e0 msgr2=0x7fe34c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:20:59.890 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 -- 172.21.15.144:0/817770072 shutdown_connections 2023-07-14T18:20:59.890 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.840+0000 7fe351c28700 1 -- 172.21.15.144:0/817770072 wait complete. 2023-07-14T18:20:59.892 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 Processor -- start 2023-07-14T18:20:59.892 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 -- start start 2023-07-14T18:20:59.892 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 0x7fe34c1000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:59.892 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 0x7fe34c0fec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0ff3b0 con 0x7fe34c105980 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c0ff520 con 0x7fe34c100740 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe351c28700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34c19fa90 con 0x7fe34c103060 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe34affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe34affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:54930/0 (socket says 172.21.15.144:54930) 2023-07-14T18:20:59.893 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe34affd700 1 -- 172.21.15.144:0/480617317 learned_addr learned my addr 172.21.15.144:0/480617317 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe34b7fe700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 0x7fe34c1000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.841+0000 7fe34bfff700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 0x7fe34c0fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 -- 172.21.15.144:0/480617317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 msgr2=0x7fe34c1000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 0x7fe34c1000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 -- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 msgr2=0x7fe34c0fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 0x7fe34c0fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 -- 172.21.15.144:0/480617317 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe344009d00 con 0x7fe34c103060 2023-07-14T18:20:59.894 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34bfff700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 0x7fe34c0fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe34affd700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe34000b530 tx=0x7fe34000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe340011830 con 0x7fe34c103060 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34c19fd70 con 0x7fe34c103060 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe340011e20 con 0x7fe34c103060 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34000f2e0 con 0x7fe34c103060 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.842+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34c1a0290 con 0x7fe34c103060 2023-07-14T18:20:59.895 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.843+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34c0fbd50 con 0x7fe34c103060 2023-07-14T18:20:59.896 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.844+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe340011990 con 0x7fe34c103060 2023-07-14T18:20:59.896 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.845+0000 7fe348ff9700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 0x7fe3340a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:20:59.896 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.845+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5430+0+0 (secure 0 0 0) 0x7fe3400cbda0 con 0x7fe34c103060 2023-07-14T18:20:59.897 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.845+0000 7fe34b7fe700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 0x7fe3340a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:20:59.897 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.845+0000 7fe34b7fe700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 0x7fe3340a3680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe33c005950 tx=0x7fe33c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:20:59.897 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:20:59.848+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe340095f90 con 0x7fe34c103060 2023-07-14T18:21:00.090 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:00.041+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe34c060a80 con 0x7fe3340a11d0 2023-07-14T18:21:00.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:21:00.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:21:00.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:00.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:21:00.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:00.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:00.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:00.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:00.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:21:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:21:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T18:21:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:00.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:01.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:01 smithi144 ceph-mon[98417]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:01.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:01 smithi144 ceph-mon[98417]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:21:01.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:01 smithi105 ceph-mon[98013]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:01.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:01 smithi105 ceph-mon[98013]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:21:01.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:01 smithi064 ceph-mon[96720]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:01.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:01 smithi064 ceph-mon[96720]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T18:21:02.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:02 smithi064 ceph-mon[96720]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:02.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:02 smithi064 ceph-mon[96720]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:21:02.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:02 smithi144 ceph-mon[98417]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:02.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:02 smithi144 ceph-mon[98417]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:21:02.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:02 smithi105 ceph-mon[98013]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:02.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:02 smithi105 ceph-mon[98013]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T18:21:03.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/3894377600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]': finished 2023-07-14T18:21:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T18:21:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:03 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/3962429550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:03.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/3894377600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]': finished 2023-07-14T18:21:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T18:21:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:03 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/3962429550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:03.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/3894377600' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]: dispatch 2023-07-14T18:21:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f001163e-106e-403e-94c6-81f02ae052ab"}]': finished 2023-07-14T18:21:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T18:21:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:03 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/3962429550' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:04.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:04 smithi144 ceph-mon[98417]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:04.558 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:04 smithi105 ceph-mon[98013]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:04.589 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:04 smithi064 ceph-mon[96720]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:05.788 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:05 smithi064 ceph-mon[96720]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:05.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:05 smithi105 ceph-mon[98013]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:05.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:05 smithi144 ceph-mon[98417]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:07.621 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:07 smithi144 ceph-mon[98417]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:07.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:07 smithi105 ceph-mon[98013]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:07.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:07 smithi064 ceph-mon[96720]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:10.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:21:10.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:10.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:09 smithi105 ceph-mon[98013]: Deploying daemon osd.8 on smithi144 2023-07-14T18:21:10.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:09 smithi105 ceph-mon[98013]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:10.339 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:21:10.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:10.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:09 smithi064 ceph-mon[96720]: Deploying daemon osd.8 on smithi144 2023-07-14T18:21:10.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:09 smithi064 ceph-mon[96720]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:10.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T18:21:10.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:10.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:09 smithi144 ceph-mon[98417]: Deploying daemon osd.8 on smithi144 2023-07-14T18:21:10.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:09 smithi144 ceph-mon[98417]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:11.564 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:11 smithi144 ceph-mon[98417]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:11.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:11 smithi105 ceph-mon[98013]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:11.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:11 smithi064 ceph-mon[96720]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:13.758 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:13 smithi144 ceph-mon[98417]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:13.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:13 smithi105 ceph-mon[98013]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:13.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:13 smithi064 ceph-mon[96720]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:14.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:14.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:14.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:15.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:15.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:15.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:15.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:15.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:15.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:15.910 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:15 smithi144 ceph-mon[98417]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:16.089 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:15 smithi064 ceph-mon[96720]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:16.308 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:15 smithi105 ceph-mon[98013]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:16.905 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.855+0000 7fe348ff9700 1 -- 172.21.15.144:0/480617317 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe34c060a80 con 0x7fe3340a11d0 2023-07-14T18:21:16.906 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 8 on host 'smithi144' 2023-07-14T18:21:16.907 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 msgr2=0x7fe3340a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:16.907 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 0x7fe3340a3680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe33c005950 tx=0x7fe33c00b410 comp rx=0 tx=0).stop 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 msgr2=0x7fe34c0fe6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe34000b530 tx=0x7fe34000b840 comp rx=0 tx=0).stop 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 shutdown_connections 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe3340a11d0 0x7fe3340a3680 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe34c105980 0x7fe34c0fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe34c100740 0x7fe34c1000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:16.908 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 --2- 172.21.15.144:0/480617317 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe34c103060 0x7fe34c0fe6f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:16.909 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.857+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 >> 172.21.15.144:0/480617317 conn(0x7fe34c0fa1e0 msgr2=0x7fe34c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:21:16.909 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.858+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 shutdown_connections 2023-07-14T18:21:16.909 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:21:16.858+0000 7fe351c28700 1 -- 172.21.15.144:0/480617317 wait complete. 2023-07-14T18:21:17.388 DEBUG:teuthology.orchestra.run.smithi144:osd.8> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T18:21:17.391 INFO:tasks.cephadm:Deploying osd.9 on smithi064 with /dev/vg_nvme/lv_1... 2023-07-14T18:21:17.391 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:21:17.422 INFO:journalctl@ceph.osd.8.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:21:17.646 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.646 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.646 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:17.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:17.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.647 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:17.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:17.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:17.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:17.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:17.974 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:21:18.042 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:21:17 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:21:17.684+0000 7f6892027640 -1 osd.8 0 log_to_monitors true 2023-07-14T18:21:18.483 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:18 smithi064 ceph-mon[96720]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:18.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:18 smithi064 ceph-mon[96720]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:18.484 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:18 smithi064 ceph-mon[96720]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:18.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:18 smithi144 ceph-mon[98417]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:18.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:18 smithi144 ceph-mon[98417]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:18.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:18 smithi144 ceph-mon[98417]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:18.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:18 smithi105 ceph-mon[98013]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:18.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:18 smithi105 ceph-mon[98013]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:18.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:18 smithi105 ceph-mon[98013]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T18:21:19.445 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:21:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:21:19.446 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2023-07-14T18:21:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2023-07-14T18:21:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111976 s, 936 MB/s 2023-07-14T18:21:19.446 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2023-07-14T18:21:19.680 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:21:19.338+0000 7f688769f700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T18:21:19.680 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:21:19.351+0000 7f6881c92700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:21:19.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-mon[98417]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:21:19.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-mon[98417]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:21:19.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:19.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-mon[98417]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:19 smithi144 ceph-mon[98417]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.808 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:19 smithi105 ceph-mon[98013]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:21:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:19 smithi105 ceph-mon[98013]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:21:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:19 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:19 smithi105 ceph-mon[98013]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:19 smithi105 ceph-mon[98013]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:19 smithi064 ceph-mon[96720]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T18:21:19.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:19 smithi064 ceph-mon[96720]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T18:21:19.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:19 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:19.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:19 smithi064 ceph-mon[96720]: from='osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.812 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:19 smithi064 ceph-mon[96720]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:21:19.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2023-07-14T18:21:20.392 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:21:20.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:20.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:20.785 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:20.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:20.787 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:21:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T18:21:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:20.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:20.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:20.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:20.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:21.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.545+0000 7f8cbbf06700 1 -- 172.21.15.64:0/2671151384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4103100 msgr2=0x7f8cb41054e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:21.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.545+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/2671151384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4103100 0x7f8cb41054e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4009990 tx=0x7f8ca4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 -- 172.21.15.64:0/2671151384 shutdown_connections 2023-07-14T18:21:21.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/2671151384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb4105a20 0x7f8cb4107e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/2671151384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb41007e0 0x7f8cb4102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/2671151384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4103100 0x7f8cb41054e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 -- 172.21.15.64:0/2671151384 >> 172.21.15.64:0/2671151384 conn(0x7f8cb40fa1e0 msgr2=0x7f8cb40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:21:21.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 -- 172.21.15.64:0/2671151384 shutdown_connections 2023-07-14T18:21:21.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 -- 172.21.15.64:0/2671151384 wait complete. 2023-07-14T18:21:21.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 Processor -- start 2023-07-14T18:21:21.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 -- start start 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.546+0000 7f8cbbf06700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb41007e0 0x7f8cb4195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb4103100 0x7f8cb4195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb419d3d0 con 0x7f8cb4105a20 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb419d510 con 0x7f8cb4103100 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb419d6a0 con 0x7f8cb41007e0 2023-07-14T18:21:21.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46288/0 (socket says 172.21.15.64:46288) 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 -- 172.21.15.64:0/3924248016 learned_addr learned my addr 172.21.15.64:0/3924248016 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cb9ca2700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb41007e0 0x7f8cb4195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cb94a1700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb4103100 0x7f8cb4195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 -- 172.21.15.64:0/3924248016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb41007e0 msgr2=0x7f8cb4195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb41007e0 0x7f8cb4195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 -- 172.21.15.64:0/3924248016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb4103100 msgr2=0x7f8cb4195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:21.600 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb4103100 0x7f8cb4195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 -- 172.21.15.64:0/3924248016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca40095f0 con 0x7f8cb4105a20 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cba4a3700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00add0 tx=0x7f8cac00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac00f070 con 0x7f8cb4105a20 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb419d980 con 0x7f8cb4105a20 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8cac007b20 con 0x7f8cb4105a20 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cac004030 con 0x7f8cb4105a20 2023-07-14T18:21:21.601 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.547+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb419de80 con 0x7f8cb4105a20 2023-07-14T18:21:21.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.548+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb40fbd50 con 0x7f8cb4105a20 2023-07-14T18:21:21.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.549+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8cac004ad0 con 0x7f8cb4105a20 2023-07-14T18:21:21.602 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.550+0000 7f8caaffd700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 0x7f8ca00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:21.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.550+0000 7f8cb9ca2700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 0x7f8ca00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:21.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.550+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f8cac0cb310 con 0x7f8cb4105a20 2023-07-14T18:21:21.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.551+0000 7f8cb9ca2700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 0x7f8ca00a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000ac50 tx=0x7f8cb000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:21:21.603 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.554+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8cac0953f0 con 0x7f8cb4105a20 2023-07-14T18:21:21.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:21:21.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi144 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi144 to 5722M 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] boot 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T18:21:21.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:21 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi144 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi144 to 5722M 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:21.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] boot 2023-07-14T18:21:21.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T18:21:21.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:21 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:21.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:21.760+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cb4060a80 con 0x7f8ca00a1150 2023-07-14T18:21:21.839 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:21:21.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:21:21.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi144 2023-07-14T18:21:21.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi144 to 5722M 2023-07-14T18:21:21.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:21.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: osd.8 [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] boot 2023-07-14T18:21:21.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T18:21:21.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:21 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T18:21:22.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:22 smithi064 ceph-mon[96720]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:21:22.654 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:22 smithi064 ceph-mon[96720]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:22.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:22.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:22.655 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:22.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:22 smithi144 ceph-mon[98417]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:21:22.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:22 smithi144 ceph-mon[98417]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:22.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:22.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:22.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:22.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:22 smithi105 ceph-mon[98013]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T18:21:22.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:22 smithi105 ceph-mon[98013]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:22.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:22.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:22.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:23.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:23 smithi144 ceph-mon[98417]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:23.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:23 smithi144 ceph-mon[98417]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:21:23.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:23 smithi105 ceph-mon[98013]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:23.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:23 smithi105 ceph-mon[98013]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:21:23.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:23 smithi064 ceph-mon[96720]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:23.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:23 smithi064 ceph-mon[96720]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T18:21:24.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:21:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]: dispatch 2023-07-14T18:21:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]': finished 2023-07-14T18:21:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T18:21:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:24 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:24.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:21:24.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:24.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]: dispatch 2023-07-14T18:21:24.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]': finished 2023-07-14T18:21:24.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T18:21:24.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:24 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:24.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T18:21:24.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:24.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]: dispatch 2023-07-14T18:21:24.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2802344961' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5efb2271-1e04-4bb2-9651-52760025b9ce"}]': finished 2023-07-14T18:21:24.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T18:21:24.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:24 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:25.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:25 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1628016864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:25.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:25 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1628016864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:25.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:25 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1628016864' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:26.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:26 smithi144 ceph-mon[98417]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2023-07-14T18:21:26.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:26 smithi105 ceph-mon[98013]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2023-07-14T18:21:26.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:26 smithi064 ceph-mon[96720]: pgmap v181: 1 pgs: 1 peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 141 KiB/s, 0 objects/s recovering 2023-07-14T18:21:27.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:27 smithi105 ceph-mon[98013]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:21:27.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:27 smithi064 ceph-mon[96720]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:21:27.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:27 smithi144 ceph-mon[98417]: pgmap v182: 1 pgs: 1 peering; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:21:30.304 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:29 smithi064 ceph-mon[96720]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-14T18:21:30.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:29 smithi105 ceph-mon[98013]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-14T18:21:30.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:29 smithi144 ceph-mon[98417]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2023-07-14T18:21:31.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:21:31.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:30 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:31.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:21:31.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:30 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:31.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T18:21:31.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:30 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:32.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:31 smithi105 ceph-mon[98013]: Deploying daemon osd.9 on smithi064 2023-07-14T18:21:32.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:31 smithi105 ceph-mon[98013]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T18:21:32.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:31 smithi064 ceph-mon[96720]: Deploying daemon osd.9 on smithi064 2023-07-14T18:21:32.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:31 smithi064 ceph-mon[96720]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T18:21:32.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:31 smithi144 ceph-mon[98417]: Deploying daemon osd.9 on smithi064 2023-07-14T18:21:32.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:31 smithi144 ceph-mon[98417]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T18:21:33.548 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:33 smithi064 ceph-mon[96720]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:21:33.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:33 smithi105 ceph-mon[98013]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:21:33.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:33 smithi144 ceph-mon[98417]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:21:36.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:36 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:36 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:36 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:36.257 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:36 smithi064 ceph-mon[96720]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:36.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:36 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:36 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:36 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:36.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:36 smithi144 ceph-mon[98417]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:36 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:36 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:36.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:36 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:36.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:36 smithi105 ceph-mon[98013]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:37.806 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:37 smithi064 ceph-mon[96720]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:37.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:37 smithi105 ceph-mon[98013]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:37.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.810+0000 7f8caaffd700 1 -- 172.21.15.64:0/3924248016 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8cb4060a80 con 0x7f8ca00a1150 2023-07-14T18:21:37.861 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 9 on host 'smithi064' 2023-07-14T18:21:37.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 msgr2=0x7f8ca00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:37.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 0x7f8ca00a3600 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8cb000ac50 tx=0x7f8cb000a380 comp rx=0 tx=0).stop 2023-07-14T18:21:37.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 msgr2=0x7f8cb419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:37.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8cac00add0 tx=0x7f8cac00c330 comp rx=0 tx=0).stop 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 shutdown_connections 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8cb41007e0 0x7f8cb4195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ca00a1150 0x7f8ca00a3600 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8cb4105a20 0x7f8cb419ce00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 --2- 172.21.15.64:0/3924248016 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cb4103100 0x7f8cb4195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 >> 172.21.15.64:0/3924248016 conn(0x7f8cb40fa1e0 msgr2=0x7f8cb4101440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 shutdown_connections 2023-07-14T18:21:37.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:21:37.813+0000 7f8cbbf06700 1 -- 172.21.15.64:0/3924248016 wait complete. 2023-07-14T18:21:37.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:37 smithi144 ceph-mon[98417]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:38.303 DEBUG:teuthology.orchestra.run.smithi064:osd.9> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T18:21:38.305 INFO:tasks.cephadm:Deploying osd.10 on smithi105 with /dev/vg_nvme/lv_1... 2023-07-14T18:21:38.306 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:21:38.341 INFO:journalctl@ceph.osd.9.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:21:38.870 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:21:39.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:38 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.090 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:21:38 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:21:38.968+0000 7fe7cdb64640 -1 osd.9 0 log_to_monitors true 2023-07-14T18:21:39.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:38 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.180 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:39.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:38 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:40.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:39 smithi064 ceph-mon[96720]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:21:40.072 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:39 smithi064 ceph-mon[96720]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:39 smithi105 ceph-mon[98013]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:21:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:39 smithi105 ceph-mon[98013]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:40.388 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:21:40.389 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:21:40.389 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2023-07-14T18:21:40.389 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2023-07-14T18:21:40.389 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111172 s, 943 MB/s 2023-07-14T18:21:40.389 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2023-07-14T18:21:40.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:39 smithi144 ceph-mon[98417]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T18:21:40.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:39 smithi144 ceph-mon[98417]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:40.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi105:vg_nvme/lv_1 2023-07-14T18:21:41.276 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:21:41.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:40 smithi105 ceph-mon[98013]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:21:41.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:40 smithi105 ceph-mon[98013]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T18:21:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:40 smithi105 ceph-mon[98013]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:21:41.340 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:21:40.971+0000 7fe7c31dc700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T18:21:41.340 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:21:40.982+0000 7fe7bcfce700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:21:41.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-mon[96720]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:21:41.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-mon[96720]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T18:21:41.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:41.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:40 smithi064 ceph-mon[96720]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:21:41.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:40 smithi144 ceph-mon[98417]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T18:21:41.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:40 smithi144 ceph-mon[98417]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T18:21:41.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:41.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:40 smithi144 ceph-mon[98417]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2023-07-14T18:21:42.298 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi064 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.300 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi064 to 4292M 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.301 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.302 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:42.302 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:41 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:21:42.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:21:42.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi064 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi064 to 4292M 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:42.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:41 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi064 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi064 to 4292M 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:42.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:21:42.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:42.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T18:21:42.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:41 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:42.452 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/1571046661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 msgr2=0x7f4ec80fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:42.452 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 --2- 172.21.15.105:0/1571046661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec80fefc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8009990 tx=0x7f4eb8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.452 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/1571046661 shutdown_connections 2023-07-14T18:21:42.452 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 --2- 172.21.15.105:0/1571046661 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 0x7f4ec8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.453 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 --2- 172.21.15.105:0/1571046661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec80fefc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.453 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 --2- 172.21.15.105:0/1571046661 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec8100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.453 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.401+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/1571046661 >> 172.21.15.105:0/1571046661 conn(0x7f4ec80fa1e0 msgr2=0x7f4ec80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:21:42.453 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.402+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/1571046661 shutdown_connections 2023-07-14T18:21:42.453 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.402+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/1571046661 wait complete. 2023-07-14T18:21:42.454 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.402+0000 7f4ecf7d3700 1 Processor -- start 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.402+0000 7f4ecf7d3700 1 -- start start 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 0x7f4ec819cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819d3c0 con 0x7f4ec8100740 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819d530 con 0x7f4ec80febb0 2023-07-14T18:21:42.455 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec819d6f0 con 0x7f4ec80ffdb0 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57422/0 (socket says 172.21.15.105:57422) 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 -- 172.21.15.105:0/4202895018 learned_addr learned my addr 172.21.15.105:0/4202895018 (peer_addr_for_me v2:172.21.15.105:0/0) 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4eccd6e700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecdd70700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 0x7f4ec819cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 -- 172.21.15.105:0/4202895018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 msgr2=0x7f4ec819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:42.456 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 -- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 msgr2=0x7f4ec819cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 0x7f4ec819cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 -- 172.21.15.105:0/4202895018 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb80095f0 con 0x7f4ec80febb0 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4eccd6e700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecd56f700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80059d0 tx=0x7f4eb8003b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb801d070 con 0x7f4ec80febb0 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/4202895018 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec819d970 con 0x7f4ec80febb0 2023-07-14T18:21:42.457 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4eb800bd70 con 0x7f4ec80febb0 2023-07-14T18:21:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8022e30 con 0x7f4ec80febb0 2023-07-14T18:21:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.403+0000 7f4ecf7d3700 1 -- 172.21.15.105:0/4202895018 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec819de60 con 0x7f4ec80febb0 2023-07-14T18:21:42.458 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.404+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea80052f0 con 0x7f4ec80febb0 2023-07-14T18:21:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.405+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4eb8003ff0 con 0x7f4ec80febb0 2023-07-14T18:21:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.406+0000 7f4ebe7fc700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 0x7f4eb40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:21:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.406+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f4eb80cd590 con 0x7f4ec80febb0 2023-07-14T18:21:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.406+0000 7f4eccd6e700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 0x7f4eb40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:21:42.459 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.407+0000 7f4eccd6e700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 0x7f4eb40a3600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4005fd0 tx=0x7f4ec4005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:21:42.460 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.410+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4eb805c780 con 0x7f4ec80febb0 2023-07-14T18:21:42.652 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:42.602+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ea8000bc0 con 0x7f4eb40a1150 2023-07-14T18:21:43.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] boot 2023-07-14T18:21:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T18:21:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:43.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:43.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:43.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] boot 2023-07-14T18:21:43.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T18:21:43.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:43.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:43.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:43.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:43.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: osd.9 [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] boot 2023-07-14T18:21:43.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T18:21:43.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T18:21:43.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:21:43.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:21:43.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:44.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:44 smithi105 ceph-mon[98013]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:44.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:44 smithi105 ceph-mon[98013]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:44.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:44 smithi064 ceph-mon[96720]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:44.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:44 smithi064 ceph-mon[96720]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:44.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:44 smithi144 ceph-mon[98417]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:21:44.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:44 smithi144 ceph-mon[98417]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:45.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T18:21:45.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/3818534555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]': finished 2023-07-14T18:21:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T18:21:45.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:45 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:21:45.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T18:21:45.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/3818534555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]': finished 2023-07-14T18:21:45.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T18:21:45.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:45 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:21:45.430 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T18:21:45.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/3818534555' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]: dispatch 2023-07-14T18:21:45.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad"}]': finished 2023-07-14T18:21:45.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T18:21:45.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:45 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:21:46.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:46 smithi064 ceph-mon[96720]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:46.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:46 smithi064 ceph-mon[96720]: from='client.? 172.21.15.105:0/1902794022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:46.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:46 smithi144 ceph-mon[98417]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:46.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:46 smithi144 ceph-mon[98417]: from='client.? 172.21.15.105:0/1902794022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:46.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:46 smithi105 ceph-mon[98013]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:46.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:46 smithi105 ceph-mon[98013]: from='client.? 172.21.15.105:0/1902794022' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:21:47.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:47 smithi105 ceph-mon[98013]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:47.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:47 smithi064 ceph-mon[96720]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:47.930 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:47 smithi144 ceph-mon[98417]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:50.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:49 smithi105 ceph-mon[98013]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:50.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:49 smithi064 ceph-mon[96720]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:50.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:49 smithi144 ceph-mon[98417]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:51.680 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:21:51.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:51.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:21:51.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:51.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T18:21:51.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:21:52.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:52 smithi105 ceph-mon[98013]: Deploying daemon osd.10 on smithi105 2023-07-14T18:21:52.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:52 smithi105 ceph-mon[98013]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:52.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:52 smithi144 ceph-mon[98417]: Deploying daemon osd.10 on smithi105 2023-07-14T18:21:52.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:52 smithi144 ceph-mon[98417]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:52.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:52 smithi064 ceph-mon[96720]: Deploying daemon osd.10 on smithi105 2023-07-14T18:21:52.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:52 smithi064 ceph-mon[96720]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:53.717 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:53 smithi105 ceph-mon[98013]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:53.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:53 smithi064 ceph-mon[96720]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:53.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:53 smithi144 ceph-mon[98417]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:55.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:55 smithi105 ceph-mon[98013]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:55.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:55 smithi064 ceph-mon[96720]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:55.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:55 smithi144 ceph-mon[98417]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:56.734 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:56 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:56.734 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:56 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:56.734 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:56 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:56.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:56 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:57.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:57.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:57.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:56 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:21:57.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:57 smithi144 ceph-mon[98417]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:57.959 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:57 smithi105 ceph-mon[98013]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:58.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:57 smithi064 ceph-mon[96720]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:21:58.319 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.270+0000 7f4ebe7fc700 1 -- 172.21.15.105:0/4202895018 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f4ea8000bc0 con 0x7f4eb40a1150 2023-07-14T18:21:58.320 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 10 on host 'smithi105' 2023-07-14T18:21:58.321 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 msgr2=0x7f4eb40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 0x7f4eb40a3600 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4005fd0 tx=0x7f4ec4005e30 comp rx=0 tx=0).stop 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 msgr2=0x7f4ec8068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4eb80059d0 tx=0x7f4eb8003b00 comp rx=0 tx=0).stop 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 shutdown_connections 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4eb40a1150 0x7f4eb40a3600 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:58.322 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4ec8100740 0x7f4ec819cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:58.323 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4ec80febb0 0x7f4ec8068cc0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:58.323 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 --2- 172.21.15.105:0/4202895018 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4ec80ffdb0 0x7f4ec819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:21:58.323 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 >> 172.21.15.105:0/4202895018 conn(0x7f4ec80fa1e0 msgr2=0x7f4ec80fc440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:21:58.323 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 shutdown_connections 2023-07-14T18:21:58.323 INFO:teuthology.orchestra.run.smithi105.stderr:2023-07-14T18:21:58.272+0000 7f4eb3fff700 1 -- 172.21.15.105:0/4202895018 wait complete. 2023-07-14T18:21:58.768 DEBUG:teuthology.orchestra.run.smithi105:osd.10> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T18:21:58.771 INFO:tasks.cephadm:Deploying osd.11 on smithi144 with /dev/vg_nvme/lv_1... 2023-07-14T18:21:58.772 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T18:21:58.801 INFO:journalctl@ceph.osd.10.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:21:59.341 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:21:59.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:21:59 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.474 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.474 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.474 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.474 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:21:59 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:21:59 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:21:59.776 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:21:59 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:21:59.559+0000 7f8ab95be640 -1 osd.10 0 log_to_monitors true 2023-07-14T18:22:00.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:00 smithi144 ceph-mon[98417]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:00.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:00 smithi144 ceph-mon[98417]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.428 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:00 smithi144 ceph-mon[98417]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:00 smithi105 ceph-mon[98013]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:00 smithi105 ceph-mon[98013]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:00 smithi105 ceph-mon[98013]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:00 smithi064 ceph-mon[96720]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:00.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:00 smithi064 ceph-mon[96720]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:00 smithi064 ceph-mon[96720]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T18:22:00.914 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T18:22:00.914 INFO:teuthology.orchestra.run.smithi144.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T18:22:00.915 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10+0 records in 2023-07-14T18:22:00.915 INFO:teuthology.orchestra.run.smithi144.stderr:10+0 records out 2023-07-14T18:22:00.915 INFO:teuthology.orchestra.run.smithi144.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011208 s, 936 MB/s 2023-07-14T18:22:00.915 INFO:teuthology.orchestra.run.smithi144.stderr:--> Zapping successful for: 2023-07-14T18:22:01.345 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:22:01.345 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T18:22:01.345 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:01.345 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.346 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.346 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:01.394 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi144:vg_nvme/lv_1 2023-07-14T18:22:01.559 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:22:01.144+0000 7f8aaec36700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T18:22:01.560 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:22:01.163+0000 7f8aa7a26700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:22:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:22:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T18:22:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:01.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:01.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T18:22:01.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T18:22:01.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:01.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: from='osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2023-07-14T18:22:01.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:01.948 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:22:02.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi105 2023-07-14T18:22:02.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi105 to 4292M 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:02 smithi144 ceph-mon[98417]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:02.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi105 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi105 to 4292M 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:02.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:02 smithi105 ceph-mon[98013]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:02.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi105 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T18:22:02.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi105 to 4292M 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:02 smithi064 ceph-mon[96720]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T18:22:03.288 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.238+0000 7fd91ca03700 1 -- 172.21.15.144:0/159979229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd918100740 msgr2=0x7fd918107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.238+0000 7fd91ca03700 1 --2- 172.21.15.144:0/159979229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd918100740 0x7fd918107000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd91000b0a0 tx=0x7fd91000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 -- 172.21.15.144:0/159979229 shutdown_connections 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 --2- 172.21.15.144:0/159979229 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd918100740 0x7fd918107000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 --2- 172.21.15.144:0/159979229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd9180febb0 0x7fd9180fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 --2- 172.21.15.144:0/159979229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180ffdb0 0x7fd918100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.289 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 -- 172.21.15.144:0/159979229 >> 172.21.15.144:0/159979229 conn(0x7fd9180fa1e0 msgr2=0x7fd9180fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:03.290 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 -- 172.21.15.144:0/159979229 shutdown_connections 2023-07-14T18:22:03.290 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 -- 172.21.15.144:0/159979229 wait complete. 2023-07-14T18:22:03.291 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 Processor -- start 2023-07-14T18:22:03.291 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.239+0000 7fd91ca03700 1 -- start start 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180febb0 0x7fd918068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd918100740 0x7fd91819ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91819d460 con 0x7fd9180febb0 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91819d5d0 con 0x7fd918100740 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91819d790 con 0x7fd9180ffdb0 2023-07-14T18:22:03.292 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55084/0 (socket says 172.21.15.144:55084) 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 -- 172.21.15.144:0/1953393024 learned_addr learned my addr 172.21.15.144:0/1953393024 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd916d9d700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd918100740 0x7fd91819ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91659c700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180febb0 0x7fd918068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 -- 172.21.15.144:0/1953393024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd918100740 msgr2=0x7fd91819ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:03.293 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd918100740 0x7fd91819ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 -- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180febb0 msgr2=0x7fd918068cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180febb0 0x7fd918068cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 -- 172.21.15.144:0/1953393024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910009d00 con 0x7fd9180ffdb0 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd915d9b700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00c9b0 tx=0x7fd90c00ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c007b00 con 0x7fd9180ffdb0 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd90c007c60 con 0x7fd9180ffdb0 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90c018660 con 0x7fd9180ffdb0 2023-07-14T18:22:03.294 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd918194460 con 0x7fd9180ffdb0 2023-07-14T18:22:03.295 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.240+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd918194960 con 0x7fd9180ffdb0 2023-07-14T18:22:03.295 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.241+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91810a230 con 0x7fd9180ffdb0 2023-07-14T18:22:03.295 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.242+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd90c00f450 con 0x7fd9180ffdb0 2023-07-14T18:22:03.296 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.243+0000 7fd9077fe700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 0x7fd9000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:03.296 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.243+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fd90c0cc260 con 0x7fd9180ffdb0 2023-07-14T18:22:03.296 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.243+0000 7fd91659c700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 0x7fd9000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:03.296 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.244+0000 7fd91659c700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 0x7fd9000a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd908005950 tx=0x7fd9080058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:03.296 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.247+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd90c095fe0 con 0x7fd9180ffdb0 2023-07-14T18:22:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:03 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:22:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:03 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:22:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:03 smithi144 ceph-mon[98417]: osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] boot 2023-07-14T18:22:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:03 smithi144 ceph-mon[98417]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T18:22:03.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:03 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:03.492 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:03.443+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd918195530 con 0x7fd9000a11d0 2023-07-14T18:22:03.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:03 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:22:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:03 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:22:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:03 smithi105 ceph-mon[98013]: osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] boot 2023-07-14T18:22:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:03 smithi105 ceph-mon[98013]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T18:22:03.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:03 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:03.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:03 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:22:03.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:03 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:22:03.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:03 smithi064 ceph-mon[96720]: osd.10 [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] boot 2023-07-14T18:22:03.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:03 smithi064 ceph-mon[96720]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T18:22:03.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:03 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T18:22:04.476 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T18:22:04.476 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:04.476 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:04.477 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:22:04.477 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:22:04.477 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:04.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:22:04.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:04.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T18:22:04.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:04.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: from='client.24379 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:22:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:22:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:05.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:05 smithi144 ceph-mon[98417]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:22:05.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:05 smithi105 ceph-mon[98013]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:22:05.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:05 smithi064 ceph-mon[96720]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T18:22:06.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/3187240855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]': finished 2023-07-14T18:22:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T18:22:06.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:06.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:22:06.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/3187240855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]': finished 2023-07-14T18:22:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T18:22:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:06.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T18:22:06.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:06.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/3187240855' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]: dispatch 2023-07-14T18:22:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599"}]': finished 2023-07-14T18:22:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T18:22:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:07.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:07 smithi105 ceph-mon[98013]: from='client.? 172.21.15.144:0/3941109045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:22:07.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:07 smithi064 ceph-mon[96720]: from='client.? 172.21.15.144:0/3941109045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:22:07.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:07 smithi144 ceph-mon[98417]: from='client.? 172.21.15.144:0/3941109045' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:22:08.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:08 smithi105 ceph-mon[98013]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:08.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:08 smithi064 ceph-mon[96720]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:08.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:08 smithi144 ceph-mon[98417]: pgmap v214: 1 pgs: 1 remapped+peering; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:10.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:09 smithi144 ceph-mon[98417]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:22:10.309 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:09 smithi105 ceph-mon[98013]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:22:10.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:09 smithi064 ceph-mon[96720]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:22:11.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:11 smithi105 ceph-mon[98013]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T18:22:11.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:11 smithi064 ceph-mon[96720]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T18:22:11.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:11 smithi144 ceph-mon[98417]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2023-07-14T18:22:12.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:22:12.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:12 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:12.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:12 smithi144 ceph-mon[98417]: Deploying daemon osd.11 on smithi144 2023-07-14T18:22:12.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:22:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:12 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:12 smithi105 ceph-mon[98013]: Deploying daemon osd.11 on smithi144 2023-07-14T18:22:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T18:22:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:12 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:12 smithi064 ceph-mon[96720]: Deploying daemon osd.11 on smithi144 2023-07-14T18:22:13.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:13 smithi144 ceph-mon[98417]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T18:22:13.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:13 smithi064 ceph-mon[96720]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T18:22:14.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:13 smithi105 ceph-mon[98013]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2023-07-14T18:22:15.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:15 smithi105 ceph-mon[98013]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:22:15.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:15 smithi064 ceph-mon[96720]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:22:15.842 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:15 smithi144 ceph-mon[98417]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2023-07-14T18:22:17.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:17.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:17.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:17 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:17.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:17 smithi144 ceph-mon[98417]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T18:22:18.059 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:17 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:17 smithi105 ceph-mon[98013]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T18:22:18.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:18.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:18.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:17 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:18.091 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:17 smithi064 ceph-mon[96720]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2023-07-14T18:22:19.605 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.554+0000 7fd9077fe700 1 -- 172.21.15.144:0/1953393024 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fd918195530 con 0x7fd9000a11d0 2023-07-14T18:22:19.606 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 11 on host 'smithi144' 2023-07-14T18:22:19.607 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 msgr2=0x7fd9000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:19.607 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 0x7fd9000a3680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd908005950 tx=0x7fd9080058e0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.607 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 msgr2=0x7fd91819a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd90c00c9b0 tx=0x7fd90c00ccc0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 shutdown_connections 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9000a11d0 0x7fd9000a3680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd918100740 0x7fd91819ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd9180febb0 0x7fd918068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 --2- 172.21.15.144:0/1953393024 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd9180ffdb0 0x7fd91819a630 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:19.608 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 >> 172.21.15.144:0/1953393024 conn(0x7fd9180fa1e0 msgr2=0x7fd9180fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:19.609 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 shutdown_connections 2023-07-14T18:22:19.609 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:19.556+0000 7fd91ca03700 1 -- 172.21.15.144:0/1953393024 wait complete. 2023-07-14T18:22:20.037 DEBUG:teuthology.orchestra.run.smithi144:osd.11> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T18:22:20.039 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T18:22:20.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T18:22:20.072 INFO:journalctl@ceph.osd.11.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:22:20.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:20.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:20.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:22:20.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:20 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:20.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:22:20.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:20.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:20 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:20.620 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:22:20.884 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:22:20 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:22:20.655+0000 7f0208453640 -1 osd.11 0 log_to_monitors true 2023-07-14T18:22:21.513 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:21 smithi144 ceph-mon[98417]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.514 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:21 smithi144 ceph-mon[98417]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:21 smithi105 ceph-mon[98013]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:21 smithi105 ceph-mon[98013]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:21 smithi064 ceph-mon[96720]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:21 smithi064 ceph-mon[96720]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T18:22:21.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 -- 172.21.15.64:0/3319362350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40febb0 msgr2=0x7f5ef40fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:21.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/3319362350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40febb0 0x7f5ef40fefc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5ee4009990 tx=0x7f5ee4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 -- 172.21.15.64:0/3319362350 shutdown_connections 2023-07-14T18:22:21.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/3319362350 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 0x7f5ef4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/3319362350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40febb0 0x7f5ef40fefc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/3319362350 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 -- 172.21.15.64:0/3319362350 >> 172.21.15.64:0/3319362350 conn(0x7f5ef40fa1e0 msgr2=0x7f5ef40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 -- 172.21.15.64:0/3319362350 shutdown_connections 2023-07-14T18:22:21.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.782+0000 7f5ef8a63700 1 -- 172.21.15.64:0/3319362350 wait complete. 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 Processor -- start 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 -- start start 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 0x7f5ef4068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 0x7f5ef419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:21.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef419d410 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef419d580 con 0x7f5ef4100740 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef8a63700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef419d740 con 0x7f5ef40febb0 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef1d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 0x7f5ef4068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef1d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46442/0 (socket says 172.21.15.64:46442) 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef1d9b700 1 -- 172.21.15.64:0/4099482020 learned_addr learned my addr 172.21.15.64:0/4099482020 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:21.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.783+0000 7f5ef2d9d700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 0x7f5ef419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 -- 172.21.15.64:0/4099482020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 msgr2=0x7f5ef4068cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 0x7f5ef4068cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 -- 172.21.15.64:0/4099482020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 msgr2=0x7f5ef419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 0x7f5ef419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 -- 172.21.15.64:0/4099482020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee40095f0 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef259c700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 0x7f5ef4068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef1d9b700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00d950 tx=0x7f5edc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:21.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc00f890 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5edc00fe80 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef419da20 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5edc00e3b0 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.784+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef419df70 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.785+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef410a220 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.787+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5edc010460 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.787+0000 7f5eeb7fe700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 0x7f5ee00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.788+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6831+0+0 (secure 0 0 0) 0x7f5edc0cc1b0 con 0x7f5ef40ffdb0 2023-07-14T18:22:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.788+0000 7f5ef259c700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 0x7f5ee00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:21.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.788+0000 7f5ef259c700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 0x7f5ee00a3680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400b5c0 tx=0x7f5ee4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:21.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.791+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5edc095e30 con 0x7f5ef40ffdb0 2023-07-14T18:22:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.989+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f5ef410a3e0 con 0x7f5ef40ffdb0 2023-07-14T18:22:22.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.989+0000 7f5eeb7fe700 1 -- 172.21.15.64:0/4099482020 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7f5edc099450 con 0x7f5ef40ffdb0 2023-07-14T18:22:22.041 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:22.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.991+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 msgr2=0x7f5ee00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.991+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 0x7f5ee00a3680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5ee400b5c0 tx=0x7f5ee4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.991+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 msgr2=0x7f5ef419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.991+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5edc00d950 tx=0x7f5edc00dc60 comp rx=0 tx=0).stop 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 shutdown_connections 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5ef40febb0 0x7f5ef4068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5ee00a11d0 0x7f5ee00a3680 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:22.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ef4100740 0x7f5ef419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:22.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 --2- 172.21.15.64:0/4099482020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ef40ffdb0 0x7f5ef419a5e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:22.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 >> 172.21.15.64:0/4099482020 conn(0x7f5ef40fa1e0 msgr2=0x7f5ef40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:22.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 shutdown_connections 2023-07-14T18:22:22.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:21.992+0000 7f5ef8a63700 1 -- 172.21.15.64:0/4099482020 wait complete. 2023-07-14T18:22:22.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:22:22.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T18:22:22.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:22.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:22.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:22 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4099482020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:22.431 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:22:22.213+0000 7f01fdacb700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T18:22:22.431 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:22:22.225+0000 7f01f98c1700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:22:22.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:22.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:22 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4099482020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:22.467 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1689358922,"num_in_osds":12,"osd_in_since":1689358925,"num_remapped_pgs":0} 2023-07-14T18:22:22.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T18:22:22.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T18:22:22.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:22.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: from='osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2023-07-14T18:22:22.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:22.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:22 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4099482020' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:23.468 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T18:22:23.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:22:23.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:22:23.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:23.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:23.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:23 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:23.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:23.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:23 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2023-07-14T18:22:23.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T18:22:23.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:23.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:22:23.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:23.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:23.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:23.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:23 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:24.046 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:22:24.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: purged_snaps scrub starts 2023-07-14T18:22:24.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: purged_snaps scrub ok 2023-07-14T18:22:24.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: Detected new or changed devices on smithi144 2023-07-14T18:22:24.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: Adjusting osd_memory_target on smithi144 to 4292M 2023-07-14T18:22:24.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] boot 2023-07-14T18:22:24.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T18:22:24.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:24.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:24 smithi105 ceph-mon[98013]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:24.590 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: purged_snaps scrub starts 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: purged_snaps scrub ok 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: Detected new or changed devices on smithi144 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: Adjusting osd_memory_target on smithi144 to 4292M 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] boot 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T18:22:24.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:24.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:24 smithi064 ceph-mon[96720]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: purged_snaps scrub starts 2023-07-14T18:22:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: purged_snaps scrub ok 2023-07-14T18:22:24.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: Detected new or changed devices on smithi144 2023-07-14T18:22:24.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: Adjusting osd_memory_target on smithi144 to 4292M 2023-07-14T18:22:24.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: osd.11 [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] boot 2023-07-14T18:22:24.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T18:22:24.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T18:22:24.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:24 smithi144 ceph-mon[98417]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T18:22:25.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 -- 172.21.15.64:0/3668462486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 msgr2=0x7f4b141013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:25.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 --2- 172.21.15.64:0/3668462486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b141013b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009990 tx=0x7f4afc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 -- 172.21.15.64:0/3668462486 shutdown_connections 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 --2- 172.21.15.64:0/3668462486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b141018f0 0x7f4b141081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 --2- 172.21.15.64:0/3668462486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b140ffd60 0x7f4b14100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 --2- 172.21.15.64:0/3668462486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b141013b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 -- 172.21.15.64:0/3668462486 >> 172.21.15.64:0/3668462486 conn(0x7f4b140fb3d0 msgr2=0x7f4b140fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 -- 172.21.15.64:0/3668462486 shutdown_connections 2023-07-14T18:22:25.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.167+0000 7f4b196e9700 1 -- 172.21.15.64:0/3668462486 wait complete. 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 Processor -- start 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 -- start start 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b140ffd60 0x7f4b140743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b141018f0 0x7f4b14072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:25.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b140736f0 con 0x7f4b14100f60 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b14073860 con 0x7f4b141018f0 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b196e9700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b1419f9e0 con 0x7f4b140ffd60 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46460/0 (socket says 172.21.15.64:46460) 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 -- 172.21.15.64:0/2911899553 learned_addr learned my addr 172.21.15.64:0/2911899553 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b137fe700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b141018f0 0x7f4b14072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b12ffd700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b140ffd60 0x7f4b140743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:25.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 -- 172.21.15.64:0/2911899553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b140ffd60 msgr2=0x7f4b140743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b140ffd60 0x7f4b140743e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 -- 172.21.15.64:0/2911899553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b141018f0 msgr2=0x7f4b14072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.168+0000 7f4b127fc700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b141018f0 0x7f4b14072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b127fc700 1 -- 172.21.15.64:0/2911899553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4afc0095f0 con 0x7f4b14100f60 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b127fc700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4afc006010 tx=0x7f4afc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc01d070 con 0x7f4b14100f60 2023-07-14T18:22:25.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4afc0041b0 con 0x7f4b14100f60 2023-07-14T18:22:25.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b1419fc60 con 0x7f4b14100f60 2023-07-14T18:22:25.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4afc022da0 con 0x7f4b14100f60 2023-07-14T18:22:25.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.169+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b141a00d0 con 0x7f4b14100f60 2023-07-14T18:22:25.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.170+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b14107440 con 0x7f4b14100f60 2023-07-14T18:22:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.171+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4afc022570 con 0x7f4b14100f60 2023-07-14T18:22:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.172+0000 7f4b0bfff700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 0x7f4b000a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.172+0000 7f4b12ffd700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 0x7f4b000a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.172+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4afc0cdd40 con 0x7f4b14100f60 2023-07-14T18:22:25.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.173+0000 7f4b12ffd700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 0x7f4b000a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b04005950 tx=0x7f4b040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:25.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.176+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4afc0979c0 con 0x7f4b14100f60 2023-07-14T18:22:25.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:25 smithi064 ceph-mon[96720]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T18:22:25.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.377+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4b1402cb30 con 0x7f4b14100f60 2023-07-14T18:22:25.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.377+0000 7f4b0bfff700 1 -- 172.21.15.64:0/2911899553 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7f4afc09afe0 con 0x7f4b14100f60 2023-07-14T18:22:25.428 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:25.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 msgr2=0x7f4b000a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:25.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 0x7f4b000a3680 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f4b04005950 tx=0x7f4b040058e0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 msgr2=0x7f4b14072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:25.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f4afc006010 tx=0x7f4afc004730 comp rx=0 tx=0).stop 2023-07-14T18:22:25.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 shutdown_connections 2023-07-14T18:22:25.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4b140ffd60 0x7f4b140743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4b000a11d0 0x7f4b000a3680 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b141018f0 0x7f4b14072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 --2- 172.21.15.64:0/2911899553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4b14100f60 0x7f4b14072a30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:25.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 >> 172.21.15.64:0/2911899553 conn(0x7f4b140fb3d0 msgr2=0x7f4b14104580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:25.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 shutdown_connections 2023-07-14T18:22:25.436 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:25.379+0000 7f4b196e9700 1 -- 172.21.15.64:0/2911899553 wait complete. 2023-07-14T18:22:25.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:25 smithi105 ceph-mon[98013]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T18:22:25.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:25 smithi144 ceph-mon[98417]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T18:22:25.792 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1689358943,"num_in_osds":12,"osd_in_since":1689358925,"num_remapped_pgs":0} 2023-07-14T18:22:25.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:22:26.356 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:22:26.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2911899553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:26.557 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:26 smithi064 ceph-mon[96720]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:26.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:26 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2911899553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:26.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:26 smithi105 ceph-mon[98013]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:26.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:26 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2911899553' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T18:22:26.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:26 smithi144 ceph-mon[98417]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:27.542 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.490+0000 7fed531e3700 1 -- 172.21.15.64:0/2968411983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c103ee0 msgr2=0x7fed4c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:27.542 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.490+0000 7fed531e3700 1 --2- 172.21.15.64:0/2968411983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c103ee0 0x7fed4c10ba60 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed4400b0d0 tx=0x7fed4400b3e0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.542 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 -- 172.21.15.64:0/2968411983 shutdown_connections 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 --2- 172.21.15.64:0/2968411983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c0fee40 0x7fed4c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 --2- 172.21.15.64:0/2968411983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c103ee0 0x7fed4c10ba60 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 --2- 172.21.15.64:0/2968411983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 -- 172.21.15.64:0/2968411983 >> 172.21.15.64:0/2968411983 conn(0x7fed4c075ca0 msgr2=0x7fed4c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 -- 172.21.15.64:0/2968411983 shutdown_connections 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 -- 172.21.15.64:0/2968411983 wait complete. 2023-07-14T18:22:27.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 Processor -- start 2023-07-14T18:22:27.544 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 -- start start 2023-07-14T18:22:27.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:27.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.491+0000 7fed531e3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c195270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:27.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed531e3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c103ee0 0x7fed4c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed531e3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c199cf0 con 0x7fed4c0fee40 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed531e3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c199e60 con 0x7fed4c0ff7e0 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed531e3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4c19a020 con 0x7fed4c103ee0 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46478/0 (socket says 172.21.15.64:46478) 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 -- 172.21.15.64:0/2671121746 learned_addr learned my addr 172.21.15.64:0/2671121746 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:27.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed51780700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c103ee0 0x7fed4c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed4bfff700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c195270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 -- 172.21.15.64:0/2671121746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c103ee0 msgr2=0x7fed4c1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c103ee0 0x7fed4c1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 -- 172.21.15.64:0/2671121746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 msgr2=0x7fed4c195270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c195270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 -- 172.21.15.64:0/2671121746 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44009d00 con 0x7fed4c0fee40 2023-07-14T18:22:27.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed4bfff700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed50f7f700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed4000c920 tx=0x7fed4000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40007b00 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed4000ce70 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4c19a300 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.492+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed40005440 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.493+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed4c1a1b60 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.494+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4c04e340 con 0x7fed4c0fee40 2023-07-14T18:22:27.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.495+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fed400055a0 con 0x7fed4c0fee40 2023-07-14T18:22:27.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.495+0000 7fed49ffb700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 0x7fed3c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:27.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.495+0000 7fed4bfff700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 0x7fed3c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:27.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.495+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fed400cbfc0 con 0x7fed4c0fee40 2023-07-14T18:22:27.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.496+0000 7fed4bfff700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 0x7fed3c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed4c196350 tx=0x7fed38008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:27.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.499+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed40095bb0 con 0x7fed4c0fee40 2023-07-14T18:22:27.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.697+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed4c1a1e80 con 0x7fed4c0fee40 2023-07-14T18:22:27.749 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.698+0000 7fed49ffb700 1 -- 172.21.15.64:0/2671121746 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18409 (secure 0 0 0) 0x7fed400991d0 con 0x7fed4c0fee40 2023-07-14T18:22:27.750 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:27.750 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":77,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:22:24.216131+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:22:27.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 msgr2=0x7fed3c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:27.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 0x7fed3c0a3680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fed4c196350 tx=0x7fed38008040 comp rx=0 tx=0).stop 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 msgr2=0x7fed4c194d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fed4000c920 tx=0x7fed4000cc30 comp rx=0 tx=0).stop 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 shutdown_connections 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fed4c103ee0 0x7fed4c1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fed3c0a11d0 0x7fed3c0a3680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed4c0fee40 0x7fed4c194d30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 --2- 172.21.15.64:0/2671121746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed4c0ff7e0 0x7fed4c195270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:27.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.700+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 >> 172.21.15.64:0/2671121746 conn(0x7fed4c075ca0 msgr2=0x7fed4c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:27.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.701+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 shutdown_connections 2023-07-14T18:22:27.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:27.701+0000 7fed531e3700 1 -- 172.21.15.64:0/2671121746 wait complete. 2023-07-14T18:22:27.774 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:27 smithi064 ceph-mon[96720]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:27.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:27 smithi105 ceph-mon[98013]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:27.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:27 smithi144 ceph-mon[98417]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:28.202 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:22:28.203 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T18:22:28.763 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:22:28.809 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:28 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2671121746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:28.840 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:28 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2671121746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:28.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:28 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2671121746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:29.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.943+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1871296961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 msgr2=0x7ff04410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:29.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.943+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1871296961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff04410ba60 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff03c00b0d0 tx=0x7ff03c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:22:29.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1871296961 shutdown_connections 2023-07-14T18:22:29.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1871296961 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff04410ba60 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:29.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1871296961 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 0x7ff0440ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:29.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1871296961 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff0441037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1871296961 >> 172.21.15.64:0/1871296961 conn(0x7ff044075ca0 msgr2=0x7ff0440760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1871296961 shutdown_connections 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1871296961 wait complete. 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 Processor -- start 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.944+0000 7ff04bfcc700 1 -- start start 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 0x7ff044196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:29.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff044197080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:29.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:29.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044197990 con 0x7ff044103ee0 2023-07-14T18:22:29.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441a2100 con 0x7ff0440fee40 2023-07-14T18:22:29.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0441a2270 con 0x7ff0440ff7e0 2023-07-14T18:22:30.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:30.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46494/0 (socket says 172.21.15.64:46494) 2023-07-14T18:22:30.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 -- 172.21.15.64:0/1728534708 learned_addr learned my addr 172.21.15.64:0/1728534708 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff049d68700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 0x7ff044196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff049567700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff044197080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:30.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 -- 172.21.15.64:0/1728534708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 msgr2=0x7ff044197080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:30.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff044197080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 -- 172.21.15.64:0/1728534708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 msgr2=0x7ff044196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 0x7ff044196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 -- 172.21.15.64:0/1728534708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c009d00 con 0x7ff044103ee0 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff049567700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff044197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04a569700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009fd0 tx=0x7ff03c009380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c00e040 con 0x7ff044103ee0 2023-07-14T18:22:30.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff03c004e40 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0441a24f0 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.945+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff03c01ba20 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.946+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0441a2960 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.947+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04404e340 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.948+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff03c01bb80 con 0x7ff044103ee0 2023-07-14T18:22:30.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.948+0000 7ff036ffd700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 0x7ff0300a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:30.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.948+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff03c0ce150 con 0x7ff044103ee0 2023-07-14T18:22:30.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.948+0000 7ff049d68700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 0x7ff0300a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:30.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.949+0000 7ff049d68700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 0x7ff0300a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff04000ac50 tx=0x7ff04000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:30.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:29.952+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff03c097dd0 con 0x7ff044103ee0 2023-07-14T18:22:30.090 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:29 smithi064 ceph-mon[96720]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:30.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.146+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff0440686f0 con 0x7ff044103ee0 2023-07-14T18:22:30.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.147+0000 7ff036ffd700 1 -- 172.21.15.64:0/1728534708 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff03c09b3f0 con 0x7ff044103ee0 2023-07-14T18:22:30.198 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:22:30.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 msgr2=0x7ff0300a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:30.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 0x7ff0300a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff04000ac50 tx=0x7ff04000a380 comp rx=0 tx=0).stop 2023-07-14T18:22:30.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 msgr2=0x7ff0441a1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:30.200 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff03c009fd0 tx=0x7ff03c009380 comp rx=0 tx=0).stop 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 shutdown_connections 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff0300a1150 0x7ff0300a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff044103ee0 0x7ff0441a1bc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0440fee40 0x7ff044196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 --2- 172.21.15.64:0/1728534708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff0440ff7e0 0x7ff044197080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 >> 172.21.15.64:0/1728534708 conn(0x7ff044075ca0 msgr2=0x7ff0440fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:30.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 shutdown_connections 2023-07-14T18:22:30.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:30.149+0000 7ff04bfcc700 1 -- 172.21.15.64:0/1728534708 wait complete. 2023-07-14T18:22:30.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:29 smithi105 ceph-mon[98013]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:30.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:29 smithi144 ceph-mon[98417]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:30.623 INFO:tasks.cephadm:Adding mds.a on smithi064 2023-07-14T18:22:30.624 INFO:tasks.cephadm:Adding mds.d on smithi064 2023-07-14T18:22:30.624 INFO:tasks.cephadm:Adding mds.g on smithi064 2023-07-14T18:22:30.624 INFO:tasks.cephadm:Adding mds.j on smithi064 2023-07-14T18:22:30.624 INFO:tasks.cephadm:Adding mds.b on smithi105 2023-07-14T18:22:30.624 INFO:tasks.cephadm:Adding mds.e on smithi105 2023-07-14T18:22:30.625 INFO:tasks.cephadm:Adding mds.h on smithi105 2023-07-14T18:22:30.625 INFO:tasks.cephadm:Adding mds.k on smithi105 2023-07-14T18:22:30.625 INFO:tasks.cephadm:Adding mds.c on smithi144 2023-07-14T18:22:30.625 INFO:tasks.cephadm:Adding mds.f on smithi144 2023-07-14T18:22:30.625 INFO:tasks.cephadm:Adding mds.i on smithi144 2023-07-14T18:22:30.626 INFO:tasks.cephadm:Adding mds.l on smithi144 2023-07-14T18:22:30.626 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l' 2023-07-14T18:22:31.197 INFO:teuthology.orchestra.run.smithi144.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:22:31.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:30 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1728534708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:31.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:30 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1728534708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:31.414 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:30 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1728534708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:32.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:31 smithi105 ceph-mon[98013]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:32.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:31 smithi064 ceph-mon[96720]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:32.403 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 -- 172.21.15.144:0/3101501314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e240fe3d0 msgr2=0x7f0e240fe820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/3101501314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e240fe3d0 0x7f0e240fe820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c00b0a0 tx=0x7f0e1c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 -- 172.21.15.144:0/3101501314 shutdown_connections 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/3101501314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e240fe3d0 0x7f0e240fe820 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/3101501314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101330 0x7f0e24101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/3101501314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e24101c80 0x7f0e240fddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.404 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 -- 172.21.15.144:0/3101501314 >> 172.21.15.144:0/3101501314 conn(0x7f0e240f99c0 msgr2=0x7f0e240fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:32.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 -- 172.21.15.144:0/3101501314 shutdown_connections 2023-07-14T18:22:32.405 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.352+0000 7f0e2a70f700 1 -- 172.21.15.144:0/3101501314 wait complete. 2023-07-14T18:22:32.406 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 Processor -- start 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 -- start start 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e240fe3d0 0x7f0e24195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101c80 0x7f0e2419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2419d5e0 con 0x7f0e240fe3d0 2023-07-14T18:22:32.407 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2419d720 con 0x7f0e24101c80 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e2a70f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2419d8b0 con 0x7f0e24101330 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e237fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e237fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.144:55210/0 (socket says 172.21.15.144:55210) 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e237fe700 1 -- 172.21.15.144:0/256961428 learned_addr learned my addr 172.21.15.144:0/256961428 (peer_addr_for_me v2:172.21.15.144:0/0) 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.353+0000 7f0e23fff700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e240fe3d0 0x7f0e24195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:32.408 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 -- 172.21.15.144:0/256961428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101c80 msgr2=0x7f0e2419d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e28cac700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101c80 0x7f0e2419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101c80 0x7f0e2419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 -- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e240fe3d0 msgr2=0x7f0e24195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e240fe3d0 0x7f0e24195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 -- 172.21.15.144:0/256961428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c009d00 con 0x7f0e24101330 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e237fe700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800b530 tx=0x7f0e1800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18011830 con 0x7f0e24101330 2023-07-14T18:22:32.409 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0e18011e20 con 0x7f0e24101330 2023-07-14T18:22:32.410 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2419db90 con 0x7f0e24101330 2023-07-14T18:22:32.410 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18010db0 con 0x7f0e24101330 2023-07-14T18:22:32.410 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.354+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2419e090 con 0x7f0e24101330 2023-07-14T18:22:32.410 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.355+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e24106ab0 con 0x7f0e24101330 2023-07-14T18:22:32.411 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.356+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0e18011990 con 0x7f0e24101330 2023-07-14T18:22:32.411 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.357+0000 7f0e217fa700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 0x7f0e0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:32.411 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.357+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0e180cc370 con 0x7f0e24101330 2023-07-14T18:22:32.411 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.357+0000 7f0e23fff700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 0x7f0e0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:32.411 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.357+0000 7f0e23fff700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 0x7f0e0c0a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e14005950 tx=0x7f0e1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:32.412 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.360+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0e18095ec0 con 0x7f0e24101330 2023-07-14T18:22:32.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:31 smithi144 ceph-mon[98417]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:32.604 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.553+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e24060c80 con 0x7f0e0c0a1150 2023-07-14T18:22:32.613 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.562+0000 7f0e217fa700 1 -- 172.21.15.144:0/256961428 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0e24060c80 con 0x7f0e0c0a1150 2023-07-14T18:22:32.613 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mds.all update... 2023-07-14T18:22:32.615 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 msgr2=0x7f0e0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:32.615 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 0x7f0e0c0a3600 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0e14005950 tx=0x7f0e1400b410 comp rx=0 tx=0).stop 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 msgr2=0x7f0e24195f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0e1800b530 tx=0x7f0e1800b840 comp rx=0 tx=0).stop 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 shutdown_connections 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0e0c0a1150 0x7f0e0c0a3600 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e24101c80 0x7f0e2419d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.616 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e240fe3d0 0x7f0e24195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.617 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 --2- 172.21.15.144:0/256961428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0e24101330 0x7f0e24195f90 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:32.617 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.564+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 >> 172.21.15.144:0/256961428 conn(0x7f0e240f99c0 msgr2=0x7f0e240fb670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:32.617 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.565+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 shutdown_connections 2023-07-14T18:22:32.617 INFO:teuthology.orchestra.run.smithi144.stderr:2023-07-14T18:22:32.565+0000 7f0e2a70f700 1 -- 172.21.15.144:0/256961428 wait complete. 2023-07-14T18:22:33.039 DEBUG:teuthology.orchestra.run.smithi064:mds.a> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T18:22:33.042 DEBUG:teuthology.orchestra.run.smithi064:mds.d> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T18:22:33.080 INFO:journalctl@ceph.mds.a.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:22:33.082 DEBUG:teuthology.orchestra.run.smithi064:mds.g> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T18:22:33.112 INFO:journalctl@ceph.mds.d.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:22:33.113 DEBUG:teuthology.orchestra.run.smithi064:mds.j> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T18:22:33.141 INFO:journalctl@ceph.mds.g.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:22:33.142 DEBUG:teuthology.orchestra.run.smithi105:mds.b> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T18:22:33.145 DEBUG:teuthology.orchestra.run.smithi105:mds.e> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T18:22:33.170 INFO:journalctl@ceph.mds.j.smithi064.stdout:-- Logs begin at Fri 2023-07-14 17:57:28 UTC. -- 2023-07-14T18:22:33.182 INFO:journalctl@ceph.mds.b.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:22:33.183 DEBUG:teuthology.orchestra.run.smithi105:mds.h> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T18:22:33.212 INFO:journalctl@ceph.mds.e.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:22:33.213 DEBUG:teuthology.orchestra.run.smithi105:mds.k> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T18:22:33.240 INFO:journalctl@ceph.mds.h.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:22:33.241 DEBUG:teuthology.orchestra.run.smithi144:mds.c> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T18:22:33.243 DEBUG:teuthology.orchestra.run.smithi144:mds.f> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T18:22:33.267 INFO:journalctl@ceph.mds.k.smithi105.stdout:-- Logs begin at Fri 2023-07-14 17:56:41 UTC. -- 2023-07-14T18:22:33.272 INFO:journalctl@ceph.mds.c.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:22:33.273 DEBUG:teuthology.orchestra.run.smithi144:mds.i> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T18:22:33.299 INFO:journalctl@ceph.mds.f.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:22:33.300 DEBUG:teuthology.orchestra.run.smithi144:mds.l> sudo journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T18:22:33.324 INFO:journalctl@ceph.mds.i.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:22:33.325 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T18:22:33.326 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:22:33.350 INFO:journalctl@ceph.mds.l.smithi144.stdout:-- Logs begin at Fri 2023-07-14 17:56:51 UTC. -- 2023-07-14T18:22:33.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 -- 172.21.15.64:0/3722466048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40feb00 msgr2=0x7f11b40fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 --2- 172.21.15.64:0/3722466048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40feb00 0x7f11b40fef10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009990 tx=0x7f11a8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 -- 172.21.15.64:0/3722466048 shutdown_connections 2023-07-14T18:22:33.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 --2- 172.21.15.64:0/3722466048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b4106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 --2- 172.21.15.64:0/3722466048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40feb00 0x7f11b40fef10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 --2- 172.21.15.64:0/3722466048 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40ffcf0 0x7f11b4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 -- 172.21.15.64:0/3722466048 >> 172.21.15.64:0/3722466048 conn(0x7f11b40fa100 msgr2=0x7f11b40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:33.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 -- 172.21.15.64:0/3722466048 shutdown_connections 2023-07-14T18:22:33.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.421+0000 7f11bace3700 1 -- 172.21.15.64:0/3722466048 wait complete. 2023-07-14T18:22:33.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 Processor -- start 2023-07-14T18:22:33.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 -- start start 2023-07-14T18:22:33.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40feb00 0x7f11b4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b419d450 con 0x7f11b40ffcf0 2023-07-14T18:22:33.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b419d5c0 con 0x7f11b4100680 2023-07-14T18:22:33.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11bace3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11b419d780 con 0x7f11b40feb00 2023-07-14T18:22:33.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46506/0 (socket says 172.21.15.64:46506) 2023-07-14T18:22:33.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 -- 172.21.15.64:0/2154052378 learned_addr learned my addr 172.21.15.64:0/2154052378 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b8a7f700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40feb00 0x7f11b4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b9280700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 -- 172.21.15.64:0/2154052378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40feb00 msgr2=0x7f11b4068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40feb00 0x7f11b4068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 -- 172.21.15.64:0/2154052378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 msgr2=0x7f11b419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b419ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b3fff700 1 -- 172.21.15.64:0/2154052378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a80095f0 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.422+0000 7f11b9280700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11b3fff700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11a000d980 tx=0x7f11a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0004220 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b419da60 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f11a0007640 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00055e0 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.423+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b419dfc0 con 0x7f11b40ffcf0 2023-07-14T18:22:33.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.424+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b41060e0 con 0x7f11b40ffcf0 2023-07-14T18:22:33.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.425+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f11a0004750 con 0x7f11b40ffcf0 2023-07-14T18:22:33.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.425+0000 7f11b1ffb700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 0x7f11a40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.425+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f11a00cc270 con 0x7f11b40ffcf0 2023-07-14T18:22:33.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.425+0000 7f11b8a7f700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 0x7f11a40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.426+0000 7f11b8a7f700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 0x7f11a40a3650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11a80098e0 tx=0x7f11a800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:33.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.430+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f11a0095e60 con 0x7f11b40ffcf0 2023-07-14T18:22:33.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.633+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11b402cae0 con 0x7f11b40ffcf0 2023-07-14T18:22:33.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.634+0000 7f11b1ffb700 1 -- 172.21.15.64:0/2154052378 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f11a0005740 con 0x7f11b40ffcf0 2023-07-14T18:22:33.686 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:33.686 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":77,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:22:24.216131+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:22:33.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 msgr2=0x7f11a40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 0x7f11a40a3650 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11a80098e0 tx=0x7f11a800b260 comp rx=0 tx=0).stop 2023-07-14T18:22:33.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 msgr2=0x7f11b419a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f11a000d980 tx=0x7f11a000dc90 comp rx=0 tx=0).stop 2023-07-14T18:22:33.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 shutdown_connections 2023-07-14T18:22:33.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f11b40feb00 0x7f11b4068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f11a40a11a0 0x7f11a40a3650 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f11b4100680 0x7f11b419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 --2- 172.21.15.64:0/2154052378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11b40ffcf0 0x7f11b419a670 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 >> 172.21.15.64:0/2154052378 conn(0x7f11b40fa100 msgr2=0x7f11b4103220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:33.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 shutdown_connections 2023-07-14T18:22:33.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.636+0000 7f11bace3700 1 -- 172.21.15.64:0/2154052378 wait complete. 2023-07-14T18:22:33.701 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:22:33.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:22:33.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 -- 172.21.15.64:0/1181121398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 msgr2=0x7f8ed8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1181121398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8106f50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000b0a0 tx=0x7f8ed000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 -- 172.21.15.64:0/1181121398 shutdown_connections 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1181121398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8106f50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1181121398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80feb10 0x7f8ed80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1181121398 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80ffd00 0x7f8ed8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 -- 172.21.15.64:0/1181121398 >> 172.21.15.64:0/1181121398 conn(0x7f8ed80fa130 msgr2=0x7f8ed80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 -- 172.21.15.64:0/1181121398 shutdown_connections 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.779+0000 7f8edf70b700 1 -- 172.21.15.64:0/1181121398 wait complete. 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 Processor -- start 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 -- start start 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80feb10 0x7f8ed8102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 0x7f8ed81010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8101dc0 con 0x7f8ed8100690 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed8101f30 con 0x7f8ed80ffd00 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8edf70b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed819fa00 con 0x7f8ed80feb10 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8eddca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8eddca8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46522/0 (socket says 172.21.15.64:46522) 2023-07-14T18:22:33.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.780+0000 7f8eddca8700 1 -- 172.21.15.64:0/1951554902 learned_addr learned my addr 172.21.15.64:0/1951554902 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8edd4a7700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80feb10 0x7f8ed8102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8edcca6700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 0x7f8ed81010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 -- 172.21.15.64:0/1951554902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80feb10 msgr2=0x7f8ed8102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80feb10 0x7f8ed8102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 -- 172.21.15.64:0/1951554902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 msgr2=0x7f8ed81010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 0x7f8ed81010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:33.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 -- 172.21.15.64:0/1951554902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed0009d00 con 0x7f8ed8100690 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8edcca6700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 0x7f8ed81010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8eddca8700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009fd0 tx=0x7f8ed000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed000e040 con 0x7f8ed8100690 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8ed0003c00 con 0x7f8ed8100690 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed819fc80 con 0x7f8ed8100690 2023-07-14T18:22:33.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed001b6c0 con 0x7f8ed8100690 2023-07-14T18:22:33.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.781+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed81a00a0 con 0x7f8ed8100690 2023-07-14T18:22:33.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.782+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed810a260 con 0x7f8ed8100690 2023-07-14T18:22:33.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.783+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ed001d470 con 0x7f8ed8100690 2023-07-14T18:22:33.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.784+0000 7f8ece7fc700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 0x7f8ec40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:33.841 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.784+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8ed00cdc50 con 0x7f8ed8100690 2023-07-14T18:22:33.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.784+0000 7f8edd4a7700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 0x7f8ec40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:33.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.784+0000 7f8edd4a7700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 0x7f8ec40a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80ffb60 tx=0x7f8ed4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:33.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.787+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ed00978d0 con 0x7f8ed8100690 2023-07-14T18:22:33.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:33.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l;count:12 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:33.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:33.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:33.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:33.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: Deploying daemon mds.l on smithi144 2023-07-14T18:22:33.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:33 smithi144 ceph-mon[98417]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:34.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.984+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8ed81a0650 con 0x7f8ed8100690 2023-07-14T18:22:34.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.984+0000 7f8ece7fc700 1 -- 172.21.15.64:0/1951554902 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8ed009aef0 con 0x7f8ed8100690 2023-07-14T18:22:34.036 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:22:34.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 msgr2=0x7f8ec40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 0x7f8ec40a3600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8ed80ffb60 tx=0x7f8ed4009260 comp rx=0 tx=0).stop 2023-07-14T18:22:34.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 msgr2=0x7f8ed8101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009fd0 tx=0x7f8ed000ba50 comp rx=0 tx=0).stop 2023-07-14T18:22:34.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 shutdown_connections 2023-07-14T18:22:34.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8ed80feb10 0x7f8ed8102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8ec40a1150 0x7f8ec40a3600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8ed8100690 0x7f8ed8101620 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 --2- 172.21.15.64:0/1951554902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8ed80ffd00 0x7f8ed81010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 >> 172.21.15.64:0/1951554902 conn(0x7f8ed80fa130 msgr2=0x7f8ed80677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:34.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 shutdown_connections 2023-07-14T18:22:34.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:33.986+0000 7f8edf70b700 1 -- 172.21.15.64:0/1951554902 wait complete. 2023-07-14T18:22:34.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:34.048 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l;count:12 2023-07-14T18:22:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:34.049 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:34.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:34.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:34.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:34.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: Deploying daemon mds.l on smithi144 2023-07-14T18:22:34.050 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:33 smithi064 ceph-mon[96720]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:34.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: Saving service mds.all spec with placement smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l;count:12 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:34.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: Deploying daemon mds.l on smithi144 2023-07-14T18:22:34.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:33 smithi105 ceph-mon[98013]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 -- 172.21.15.64:0/1026593958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 msgr2=0x7fb1740fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 --2- 172.21.15.64:0/1026593958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb1740fef10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009990 tx=0x7fb15c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 -- 172.21.15.64:0/1026593958 shutdown_connections 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 --2- 172.21.15.64:0/1026593958 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 0x7fb174106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 --2- 172.21.15.64:0/1026593958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb1740fef10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 --2- 172.21.15.64:0/1026593958 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb174100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 -- 172.21.15.64:0/1026593958 >> 172.21.15.64:0/1026593958 conn(0x7fb1740fa100 msgr2=0x7fb1740fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 -- 172.21.15.64:0/1026593958 shutdown_connections 2023-07-14T18:22:34.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.126+0000 7fb1791df700 1 -- 172.21.15.64:0/1026593958 wait complete. 2023-07-14T18:22:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 Processor -- start 2023-07-14T18:22:34.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 -- start start 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb17419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 0x7fb17419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17419d370 con 0x7fb1740feb00 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17419d4e0 con 0x7fb174100680 2023-07-14T18:22:34.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb1791df700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17419d6a0 con 0x7fb1740ffcf0 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46532/0 (socket says 172.21.15.64:46532) 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb17259c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb17419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 -- 172.21.15.64:0/4051987611 learned_addr learned my addr 172.21.15.64:0/4051987611 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb17359e700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 0x7fb17419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 -- 172.21.15.64:0/4051987611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 msgr2=0x7fb17419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb17419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 -- 172.21.15.64:0/4051987611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 msgr2=0x7fb17419cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 0x7fb17419cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb172d9d700 1 -- 172.21.15.64:0/4051987611 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c0095f0 con 0x7fb1740feb00 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.127+0000 7fb17259c700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb17419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb172d9d700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb15c000c00 tx=0x7fb15c0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c01d070 con 0x7fb1740feb00 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb15c00be50 con 0x7fb1740feb00 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17419d920 con 0x7fb1740feb00 2023-07-14T18:22:34.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00f460 con 0x7fb1740feb00 2023-07-14T18:22:34.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.128+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17419dda0 con 0x7fb1740feb00 2023-07-14T18:22:34.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.129+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb174105ab0 con 0x7fb1740feb00 2023-07-14T18:22:34.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.130+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb15c022670 con 0x7fb1740feb00 2023-07-14T18:22:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.130+0000 7fb16bfff700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 0x7fb1600a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.131+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb15c0cd9e0 con 0x7fb1740feb00 2023-07-14T18:22:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.131+0000 7fb17259c700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 0x7fb1600a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.131+0000 7fb17259c700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 0x7fb1600a36d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1740680a0 tx=0x7fb164009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:34.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.135+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb15c097660 con 0x7fb1740feb00 2023-07-14T18:22:34.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.379+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17402c8e0 con 0x7fb1740feb00 2023-07-14T18:22:34.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.380+0000 7fb16bfff700 1 -- 172.21.15.64:0/4051987611 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7fb15c09ac80 con 0x7fb1740feb00 2023-07-14T18:22:34.431 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:34.431 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T18:22:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 msgr2=0x7fb1600a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 0x7fb1600a36d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb1740680a0 tx=0x7fb164009260 comp rx=0 tx=0).stop 2023-07-14T18:22:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 msgr2=0x7fb174068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb15c000c00 tx=0x7fb15c0047a0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 shutdown_connections 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb1600a1220 0x7fb1600a36d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb174100680 0x7fb17419cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1740feb00 0x7fb174068330 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 --2- 172.21.15.64:0/4051987611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb1740ffcf0 0x7fb17419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 >> 172.21.15.64:0/4051987611 conn(0x7fb1740fa100 msgr2=0x7fb1740fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 shutdown_connections 2023-07-14T18:22:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:34.382+0000 7fb1791df700 1 -- 172.21.15.64:0/4051987611 wait complete. 2023-07-14T18:22:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 1 2023-07-14T18:22:34.648 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 systemd[1]: Starting Ceph mds.l for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:34.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:34 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2154052378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:34.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:34 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1951554902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:34.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:34 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4051987611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:34.931 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 podman[123719]: 2023-07-14T18:22:34.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:34 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2154052378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:34.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:34 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1951554902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:34.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:34 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4051987611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:35.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:34 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2154052378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:22:35.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:34 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1951554902' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:22:35.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:34 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4051987611' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:35.181 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 ceph-mds[123758]: -- 172.21.15.144:0/2286810829 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557c1c4051e0 con 0x557c1c3db000 2023-07-14T18:22:35.182 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 bash[123719]: d8697520074bf187260b187bf606a4f0920c930e33b9c53de54a137cf99baa76 2023-07-14T18:22:35.182 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-l[123734]: starting mds.l at 2023-07-14T18:22:35.182 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:22:34 smithi144 systemd[1]: Started Ceph mds.l for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:36.178 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6832/4249483747,v1:172.21.15.144:6833/4249483747] up:boot 2023-07-14T18:22:36.178 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: fsmap 1 up:standby 2023-07-14T18:22:36.178 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:22:36.178 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: Deploying daemon mds.k on smithi105 2023-07-14T18:22:36.179 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:35 smithi105 ceph-mon[98013]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:36.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6832/4249483747,v1:172.21.15.144:6833/4249483747] up:boot 2023-07-14T18:22:36.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: fsmap 1 up:standby 2023-07-14T18:22:36.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:22:36.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:36.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: Deploying daemon mds.k on smithi105 2023-07-14T18:22:36.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:35 smithi064 ceph-mon[96720]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:36.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6832/4249483747,v1:172.21.15.144:6833/4249483747] up:boot 2023-07-14T18:22:36.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: fsmap 1 up:standby 2023-07-14T18:22:36.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T18:22:36.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: Deploying daemon mds.k on smithi105 2023-07-14T18:22:36.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:35 smithi144 ceph-mon[98417]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:36.446 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:36.560 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 systemd[1]: Starting Ceph mds.k for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 -- 172.21.15.64:0/1993882577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 msgr2=0x7f9e080fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:36.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1993882577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e080fef10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc009990 tx=0x7f9dfc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 -- 172.21.15.64:0/1993882577 shutdown_connections 2023-07-14T18:22:36.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1993882577 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e08106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1993882577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e080fef10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1993882577 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e080ffcf0 0x7f9e08100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 -- 172.21.15.64:0/1993882577 >> 172.21.15.64:0/1993882577 conn(0x7f9e080fa100 msgr2=0x7f9e080fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:36.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 -- 172.21.15.64:0/1993882577 shutdown_connections 2023-07-14T18:22:36.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 -- 172.21.15.64:0/1993882577 wait complete. 2023-07-14T18:22:36.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.537+0000 7f9e0edab700 1 Processor -- start 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 -- start start 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e080ffcf0 0x7f9e0819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e0819cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819d390 con 0x7f9e080feb00 2023-07-14T18:22:36.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819d500 con 0x7f9e08100680 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0819d6c0 con 0x7f9e080ffcf0 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46546/0 (socket says 172.21.15.64:46546) 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 -- 172.21.15.64:0/1602736157 learned_addr learned my addr 172.21.15.64:0/1602736157 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0d348700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e0819cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 -- 172.21.15.64:0/1602736157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e080ffcf0 msgr2=0x7f9e0819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:36.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e080ffcf0 0x7f9e0819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 -- 172.21.15.64:0/1602736157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 msgr2=0x7f9e0819cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e0819cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 -- 172.21.15.64:0/1602736157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc0095f0 con 0x7f9e080feb00 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0d348700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e0819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0cb47700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc000c00 tx=0x7f9dfc0040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc01d070 con 0x7f9e080feb00 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9dfc0041d0 con 0x7f9e080feb00 2023-07-14T18:22:36.597 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc00f6c0 con 0x7f9e080feb00 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.538+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0819d940 con 0x7f9e080feb00 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.539+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0819dd10 con 0x7f9e080feb00 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.540+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e08106110 con 0x7f9e080feb00 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.541+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9dfc022470 con 0x7f9e080feb00 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.541+0000 7f9e05ffb700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 0x7f9df80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.541+0000 7f9e07fff700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 0x7f9df80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:36.598 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.542+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9dfc0ce3c0 con 0x7f9e080feb00 2023-07-14T18:22:36.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.542+0000 7f9e07fff700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 0x7f9df80a3650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9df4005950 tx=0x7f9df40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:36.599 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.546+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9dfc098040 con 0x7f9e080feb00 2023-07-14T18:22:36.841 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.790+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e0802c8e0 con 0x7f9e080feb00 2023-07-14T18:22:36.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.791+0000 7f9e05ffb700 1 -- 172.21.15.64:0/1602736157 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f9dfc09b660 con 0x7f9e080feb00 2023-07-14T18:22:36.842 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:36.842 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T18:22:36.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 msgr2=0x7f9df80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:36.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 0x7f9df80a3650 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9df4005950 tx=0x7f9df40058e0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 msgr2=0x7f9e08068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:36.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc000c00 tx=0x7f9dfc0040b0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 shutdown_connections 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9df80a11a0 0x7f9df80a3650 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9e08100680 0x7f9e0819cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9e080feb00 0x7f9e08068330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 --2- 172.21.15.64:0/1602736157 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9e080ffcf0 0x7f9e0819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:36.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 >> 172.21.15.64:0/1602736157 conn(0x7f9e080fa100 msgr2=0x7f9e08103250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:36.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 shutdown_connections 2023-07-14T18:22:36.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:36.793+0000 7f9e0edab700 1 -- 172.21.15.64:0/1602736157 wait complete. 2023-07-14T18:22:36.846 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 2 2023-07-14T18:22:36.999 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 podman[122219]: 2023-07-14T18:22:36.999 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 ceph-mds[122259]: -- 172.21.15.105:0/523095978 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e14f251e0 con 0x557e14efa800 2023-07-14T18:22:36.999 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 bash[122219]: 6a43c89448fcae21ddfec39af216d1490be1e203b38ca92bb47962f8e9a894bd 2023-07-14T18:22:36.999 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-k[122235]: starting mds.k at 2023-07-14T18:22:37.310 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:22:36 smithi105 systemd[1]: Started Ceph mds.k for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:37.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:36 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1602736157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:37.340 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:36 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1602736157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:37.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:36 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1602736157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6832/3187736260,v1:172.21.15.105:6833/3187736260] up:boot 2023-07-14T18:22:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: fsmap 2 up:standby 2023-07-14T18:22:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:22:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: Deploying daemon mds.j on smithi064 2023-07-14T18:22:38.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:37 smithi105 ceph-mon[98013]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:38.326 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6832/3187736260,v1:172.21.15.105:6833/3187736260] up:boot 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: fsmap 2 up:standby 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:38.327 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:38.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:38.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: Deploying daemon mds.j on smithi064 2023-07-14T18:22:38.328 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:37 smithi064 ceph-mon[96720]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:38.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6832/3187736260,v1:172.21.15.105:6833/3187736260] up:boot 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: fsmap 2 up:standby 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:38.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:38.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:38.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: Deploying daemon mds.j on smithi064 2023-07-14T18:22:38.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:37 smithi144 ceph-mon[98417]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:38.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:38.886 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:38 smithi064 systemd[1]: Starting Ceph mds.j for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:38.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 -- 172.21.15.64:0/1454740364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc1006e0 msgr2=0x7f4cdc102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:38.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 --2- 172.21.15.64:0/1454740364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc102ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0009990 tx=0x7f4cd0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:38.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 -- 172.21.15.64:0/1454740364 shutdown_connections 2023-07-14T18:22:38.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 --2- 172.21.15.64:0/1454740364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc105920 0x7f4cdc107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:38.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 --2- 172.21.15.64:0/1454740364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc102ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:38.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 --2- 172.21.15.64:0/1454740364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc103000 0x7f4cdc1053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:38.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 -- 172.21.15.64:0/1454740364 >> 172.21.15.64:0/1454740364 conn(0x7f4cdc0fa100 msgr2=0x7f4cdc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:38.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 -- 172.21.15.64:0/1454740364 shutdown_connections 2023-07-14T18:22:38.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.942+0000 7f4ce320f700 1 -- 172.21.15.64:0/1454740364 wait complete. 2023-07-14T18:22:38.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 Processor -- start 2023-07-14T18:22:38.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 -- start start 2023-07-14T18:22:38.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc100050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:38.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:38.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 0x7f4cdc0febd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:38.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0ff370 con 0x7f4cdc103000 2023-07-14T18:22:38.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc0ff4e0 con 0x7f4cdc105920 2023-07-14T18:22:38.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce320f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cdc19f7f0 con 0x7f4cdc1006e0 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46562/0 (socket says 172.21.15.64:46562) 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 -- 172.21.15.64:0/2493448982 learned_addr learned my addr 172.21.15.64:0/2493448982 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce17ac700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 0x7f4cdc0febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce0fab700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc100050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 -- 172.21.15.64:0/2493448982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc1006e0 msgr2=0x7f4cdc100050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:38.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc100050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 -- 172.21.15.64:0/2493448982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 msgr2=0x7f4cdc0febd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 0x7f4cdc0febd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 -- 172.21.15.64:0/2493448982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00095f0 con 0x7f4cdc103000 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4ce17ac700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 0x7f4cdc0febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.943+0000 7f4cdbfff700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800c9b0 tx=0x7f4cc800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:39.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.944+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8007b00 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.944+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4cc8007c60 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.944+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cdc19fad0 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.944+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cc8018660 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.944+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cdc1a0030 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.945+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cdc10b260 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.946+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4cc800f450 con 0x7f4cdc103000 2023-07-14T18:22:39.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.946+0000 7f4cd9ffb700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 0x7f4ccc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:39.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.947+0000 7f4ce0fab700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 0x7f4ccc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:39.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.947+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4cc80cc460 con 0x7f4cdc103000 2023-07-14T18:22:39.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.947+0000 7f4ce0fab700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 0x7f4ccc0a36d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b5c0 tx=0x7f4cd0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:39.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:38.951+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4cc8096050 con 0x7f4cdc103000 2023-07-14T18:22:39.198 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:38 smithi064 podman[130066]: 2023-07-14T18:22:39.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.200+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cdc10b450 con 0x7f4cdc103000 2023-07-14T18:22:39.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.201+0000 7f4cd9ffb700 1 -- 172.21.15.64:0/2493448982 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2020 (secure 0 0 0) 0x7f4cc8099670 con 0x7f4cdc103000 2023-07-14T18:22:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:39.253 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T18:22:39.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 msgr2=0x7f4ccc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:39.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 0x7f4ccc0a36d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4cd000b5c0 tx=0x7f4cd0005fb0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 msgr2=0x7f4cdc0fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4cc800c9b0 tx=0x7f4cc800cd70 comp rx=0 tx=0).stop 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 shutdown_connections 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4cdc1006e0 0x7f4cdc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4ccc0a1220 0x7f4ccc0a36d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cdc105920 0x7f4cdc0febd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 --2- 172.21.15.64:0/2493448982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4cdc103000 0x7f4cdc0fe690 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:39.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 >> 172.21.15.64:0/2493448982 conn(0x7f4cdc0fa100 msgr2=0x7f4cdc101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:39.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 shutdown_connections 2023-07-14T18:22:39.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:39.203+0000 7f4ce320f700 1 -- 172.21.15.64:0/2493448982 wait complete. 2023-07-14T18:22:39.256 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 3 2023-07-14T18:22:39.591 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:39 smithi064 bash[130066]: ab5a6a847d36c6d033a3a3c47ad2455515a5076ad8193b96983a8d9ac1428139 2023-07-14T18:22:39.591 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mds[130149]: -- 172.21.15.64:0/1480037378 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5599b55b71e0 con 0x5599b558cc00 2023-07-14T18:22:39.591 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: starting mds.j at 2023-07-14T18:22:39.591 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:22:39 smithi064 systemd[1]: Started Ceph mds.j for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:40.291 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2493448982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:40.291 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.291 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.291 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.291 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:40.292 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:40.292 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:40.292 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: Deploying daemon mds.d on smithi064 2023-07-14T18:22:40.292 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:39 smithi064 ceph-mon[96720]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2493448982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:40.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:40.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:40.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:40.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: Deploying daemon mds.d on smithi064 2023-07-14T18:22:40.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:39 smithi105 ceph-mon[98013]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:40.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2493448982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:40.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:40.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:40.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:40.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:40.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: Deploying daemon mds.d on smithi064 2023-07-14T18:22:40.434 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:39 smithi144 ceph-mon[98417]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:41.035 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:40 smithi064 systemd[1]: Starting Ceph mds.d for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:41.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:41.296 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:41 smithi064 podman[130450]: 2023-07-14T18:22:41.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:40 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6834/2756017867,v1:172.21.15.64:6835/2756017867] up:boot 2023-07-14T18:22:41.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:40 smithi064 ceph-mon[96720]: fsmap 3 up:standby 2023-07-14T18:22:41.297 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:40 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:22:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:40 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6834/2756017867,v1:172.21.15.64:6835/2756017867] up:boot 2023-07-14T18:22:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:40 smithi105 ceph-mon[98013]: fsmap 3 up:standby 2023-07-14T18:22:41.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:40 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:22:41.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 -- 172.21.15.64:0/1142111186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c100e20 msgr2=0x7fde8c101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 --2- 172.21.15.64:0/1142111186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c100e20 0x7fde8c101270 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde80009990 tx=0x7fde80009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 -- 172.21.15.64:0/1142111186 shutdown_connections 2023-07-14T18:22:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 --2- 172.21.15.64:0/1142111186 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde8c1017b0 0x7fde8c108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 --2- 172.21.15.64:0/1142111186 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde8c0ffc30 0x7fde8c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 --2- 172.21.15.64:0/1142111186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c100e20 0x7fde8c101270 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 -- 172.21.15.64:0/1142111186 >> 172.21.15.64:0/1142111186 conn(0x7fde8c0fb210 msgr2=0x7fde8c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:41.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 -- 172.21.15.64:0/1142111186 shutdown_connections 2023-07-14T18:22:41.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.358+0000 7fde93419700 1 -- 172.21.15.64:0/1142111186 wait complete. 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 Processor -- start 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 -- start start 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde8c100e20 0x7fde8c195f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde8c1017b0 0x7fde8c19cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:41.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19d5b0 con 0x7fde8c0ffc30 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19d6f0 con 0x7fde8c100e20 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde93419700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8c19d880 con 0x7fde8c1017b0 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde911b5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde911b5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46596/0 (socket says 172.21.15.64:46596) 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.359+0000 7fde911b5700 1 -- 172.21.15.64:0/1937148177 learned_addr learned my addr 172.21.15.64:0/1937148177 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 -- 172.21.15.64:0/1937148177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde8c1017b0 msgr2=0x7fde8c19cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde8c1017b0 0x7fde8c19cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 -- 172.21.15.64:0/1937148177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde8c100e20 msgr2=0x7fde8c195f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde8c100e20 0x7fde8c195f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 -- 172.21.15.64:0/1937148177 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde800095f0 con 0x7fde8c0ffc30 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde911b5700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde8400b9c0 tx=0x7fde8400bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8400ca30 con 0x7fde8c0ffc30 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fde8400cb90 con 0x7fde8c0ffc30 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c19db60 con 0x7fde8c0ffc30 2023-07-14T18:22:41.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde840125a0 con 0x7fde8c0ffc30 2023-07-14T18:22:41.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.360+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c19e140 con 0x7fde8c0ffc30 2023-07-14T18:22:41.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.361+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c10b420 con 0x7fde8c0ffc30 2023-07-14T18:22:41.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.362+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fde8400cd00 con 0x7fde8c0ffc30 2023-07-14T18:22:41.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.362+0000 7fde8a7fc700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 0x7fde740a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:41.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.363+0000 7fde909b4700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 0x7fde740a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:41.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.363+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fde84006e40 con 0x7fde8c0ffc30 2023-07-14T18:22:41.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.363+0000 7fde909b4700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 0x7fde740a36d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fde800095c0 tx=0x7fde80009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:41.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.366+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fde84095980 con 0x7fde8c0ffc30 2023-07-14T18:22:41.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:40 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6834/2756017867,v1:172.21.15.64:6835/2756017867] up:boot 2023-07-14T18:22:41.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:40 smithi144 ceph-mon[98417]: fsmap 3 up:standby 2023-07-14T18:22:41.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:40 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:22:41.591 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:41 smithi064 bash[130450]: fc7cc862323b27bbf85a2ecd7b1c77d1b958a76b72e2e4868e041b2c0808062c 2023-07-14T18:22:41.591 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:41 smithi064 ceph-mds[130513]: -- 172.21.15.64:0/2353597051 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b5112211e0 con 0x55b5111f6c00 2023-07-14T18:22:41.591 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:41 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: starting mds.d at 2023-07-14T18:22:41.592 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:22:41 smithi064 systemd[1]: Started Ceph mds.d for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:41.668 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.617+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde8c106bb0 con 0x7fde8c0ffc30 2023-07-14T18:22:41.669 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.617+0000 7fde8a7fc700 1 -- 172.21.15.64:0/1937148177 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2777 (secure 0 0 0) 0x7fde84098fa0 con 0x7fde8c0ffc30 2023-07-14T18:22:41.669 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:41.669 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.619+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 msgr2=0x7fde740a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.619+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 0x7fde740a36d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fde800095c0 tx=0x7fde80009510 comp rx=0 tx=0).stop 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 msgr2=0x7fde8c195a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde8400b9c0 tx=0x7fde8400bd80 comp rx=0 tx=0).stop 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 shutdown_connections 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fde8c1017b0 0x7fde8c19cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fde740a1220 0x7fde740a36d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fde8c0ffc30 0x7fde8c195a30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 --2- 172.21.15.64:0/1937148177 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fde8c100e20 0x7fde8c195f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 >> 172.21.15.64:0/1937148177 conn(0x7fde8c0fb210 msgr2=0x7fde8c0fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 shutdown_connections 2023-07-14T18:22:41.672 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:41.620+0000 7fde93419700 1 -- 172.21.15.64:0/1937148177 wait complete. 2023-07-14T18:22:41.673 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 4 2023-07-14T18:22:42.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:42.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: Deploying daemon mds.e on smithi105 2023-07-14T18:22:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1937148177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6836/1035088143,v1:172.21.15.64:6837/1035088143] up:boot 2023-07-14T18:22:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: fsmap 4 up:standby 2023-07-14T18:22:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:42 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:22:42.710 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.710 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.710 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: Deploying daemon mds.e on smithi105 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1937148177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:42.711 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6836/1035088143,v1:172.21.15.64:6837/1035088143] up:boot 2023-07-14T18:22:42.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: fsmap 4 up:standby 2023-07-14T18:22:42.712 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:42 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:22:42.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: Deploying daemon mds.e on smithi105 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:42.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1937148177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:42.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6836/1035088143,v1:172.21.15.64:6837/1035088143] up:boot 2023-07-14T18:22:42.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: fsmap 4 up:standby 2023-07-14T18:22:42.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:42 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:22:43.060 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:42 smithi105 systemd[1]: Starting Ceph mds.e for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:43.468 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:43 smithi105 podman[122558]: 2023-07-14T18:22:43.684 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:43.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-mon[98013]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:43.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6834/2365162845,v1:172.21.15.105:6835/2365162845] up:boot 2023-07-14T18:22:43.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-mon[98013]: fsmap 5 up:standby 2023-07-14T18:22:43.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:22:43.811 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-mds[122598]: -- 172.21.15.105:0/3080215377 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563d154291e0 con 0x563d153ff000 2023-07-14T18:22:43.811 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:43 smithi105 bash[122558]: 0afefd87911e0677cf00d34939415a9da6de7fb89e16cfc1ae3f0eb393d47ce0 2023-07-14T18:22:43.811 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:43 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: starting mds.e at 2023-07-14T18:22:43.811 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:22:43 smithi105 systemd[1]: Started Ceph mds.e for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 -- 172.21.15.64:0/2712702193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540ffd10 msgr2=0x7fe654100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 --2- 172.21.15.64:0/2712702193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540ffd10 0x7fe654100160 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009990 tx=0x7fe63c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 -- 172.21.15.64:0/2712702193 shutdown_connections 2023-07-14T18:22:43.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 --2- 172.21.15.64:0/2712702193 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe654106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 --2- 172.21.15.64:0/2712702193 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540feb20 0x7fe6540fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 --2- 172.21.15.64:0/2712702193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540ffd10 0x7fe654100160 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 -- 172.21.15.64:0/2712702193 >> 172.21.15.64:0/2712702193 conn(0x7fe6540fa100 msgr2=0x7fe6540fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 -- 172.21.15.64:0/2712702193 shutdown_connections 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 -- 172.21.15.64:0/2712702193 wait complete. 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.768+0000 7fe659f13700 1 Processor -- start 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 -- start start 2023-07-14T18:22:43.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540ffd10 0x7fe65419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe65419cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419d3b0 con 0x7fe6540feb20 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419d520 con 0x7fe6540ffd10 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65419d6e0 con 0x7fe6541006a0 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:43.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46612/0 (socket says 172.21.15.64:46612) 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 -- 172.21.15.64:0/535324015 learned_addr learned my addr 172.21.15.64:0/535324015 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe652ffd700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540ffd10 0x7fe65419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe653fff700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe65419cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 -- 172.21.15.64:0/535324015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 msgr2=0x7fe65419cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe65419cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 -- 172.21.15.64:0/535324015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540ffd10 msgr2=0x7fe65419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540ffd10 0x7fe65419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 -- 172.21.15.64:0/535324015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c0095f0 con 0x7fe6540feb20 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe653fff700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe65419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe6537fe700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe64400d980 tx=0x7fe64400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644004020 con 0x7fe6540feb20 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe644009730 con 0x7fe6540feb20 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.769+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65419d9c0 con 0x7fe6540feb20 2023-07-14T18:22:43.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.770+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe644003a00 con 0x7fe6540feb20 2023-07-14T18:22:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.770+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65419ded0 con 0x7fe6540feb20 2023-07-14T18:22:43.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.770+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65410a1f0 con 0x7fe6540feb20 2023-07-14T18:22:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.772+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe6440098a0 con 0x7fe6540feb20 2023-07-14T18:22:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.772+0000 7fe650ff9700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 0x7fe6400a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.772+0000 7fe652ffd700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 0x7fe6400a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:43.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.772+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe6440cc1a0 con 0x7fe6540feb20 2023-07-14T18:22:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.773+0000 7fe652ffd700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 0x7fe6400a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe63c006010 tx=0x7fe63c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:43.828 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:43.776+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe644095d90 con 0x7fe6540feb20 2023-07-14T18:22:43.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:43 smithi064 ceph-mon[96720]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:43.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:43 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6834/2365162845,v1:172.21.15.105:6835/2365162845] up:boot 2023-07-14T18:22:43.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:43 smithi064 ceph-mon[96720]: fsmap 5 up:standby 2023-07-14T18:22:43.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:43 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:22:43.931 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:43 smithi144 ceph-mon[98417]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:43.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:43 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6834/2365162845,v1:172.21.15.105:6835/2365162845] up:boot 2023-07-14T18:22:43.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:43 smithi144 ceph-mon[98417]: fsmap 5 up:standby 2023-07-14T18:22:43.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:43 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:22:44.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.022+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe654105a40 con 0x7fe6540feb20 2023-07-14T18:22:44.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.023+0000 7fe650ff9700 1 -- 172.21.15.64:0/535324015 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4294 (secure 0 0 0) 0x7fe6440993b0 con 0x7fe6540feb20 2023-07-14T18:22:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:44.075 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T18:22:44.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 msgr2=0x7fe6400a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:44.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 0x7fe6400a3650 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe63c006010 tx=0x7fe63c00b260 comp rx=0 tx=0).stop 2023-07-14T18:22:44.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 msgr2=0x7fe654069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:44.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe64400d980 tx=0x7fe64400dc90 comp rx=0 tx=0).stop 2023-07-14T18:22:44.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 shutdown_connections 2023-07-14T18:22:44.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6541006a0 0x7fe65419cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6400a11a0 0x7fe6400a3650 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6540feb20 0x7fe654069340 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 --2- 172.21.15.64:0/535324015 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6540ffd10 0x7fe65419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 >> 172.21.15.64:0/535324015 conn(0x7fe6540fa100 msgr2=0x7fe6540fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 shutdown_connections 2023-07-14T18:22:44.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:44.025+0000 7fe659f13700 1 -- 172.21.15.64:0/535324015 wait complete. 2023-07-14T18:22:44.079 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 6 2023-07-14T18:22:44.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:44.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:44.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:44.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: Deploying daemon mds.f on smithi144 2023-07-14T18:22:44.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:44 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/535324015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:44.926 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:44.927 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: Deploying daemon mds.f on smithi144 2023-07-14T18:22:44.928 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:44 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/535324015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:45.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:45.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:45.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: Deploying daemon mds.f on smithi144 2023-07-14T18:22:45.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:44 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/535324015' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:45.178 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 systemd[1]: Starting Ceph mds.f for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:45.763 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:45 smithi144 ceph-mon[98417]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:45.763 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 podman[124061]: 2023-07-14T18:22:45.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:45 smithi064 ceph-mon[96720]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:46.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:45 smithi105 ceph-mon[98013]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:46.090 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:46.181 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 bash[124061]: 3779737864b34e98b072d74adcfbdc60ae0c3881987e4a1ae8d2a84683b7c38c 2023-07-14T18:22:46.182 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 ceph-mds[124101]: -- 172.21.15.144:0/4243176034 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b822d731e0 con 0x55b822d49000 2023-07-14T18:22:46.182 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: starting mds.f at 2023-07-14T18:22:46.182 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:22:45 smithi144 systemd[1]: Started Ceph mds.f for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 -- 172.21.15.64:0/320569033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 msgr2=0x7f3b2c0fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:46.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 --2- 172.21.15.64:0/320569033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c0fef10 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009990 tx=0x7f3b1c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 -- 172.21.15.64:0/320569033 shutdown_connections 2023-07-14T18:22:46.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 --2- 172.21.15.64:0/320569033 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 0x7f3b2c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 --2- 172.21.15.64:0/320569033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c0fef10 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 --2- 172.21.15.64:0/320569033 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 0x7f3b2c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 -- 172.21.15.64:0/320569033 >> 172.21.15.64:0/320569033 conn(0x7f3b2c0fa100 msgr2=0x7f3b2c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:46.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 -- 172.21.15.64:0/320569033 shutdown_connections 2023-07-14T18:22:46.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.178+0000 7f3b31da0700 1 -- 172.21.15.64:0/320569033 wait complete. 2023-07-14T18:22:46.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 Processor -- start 2023-07-14T18:22:46.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 -- start start 2023-07-14T18:22:46.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:46.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 0x7f3b2c19a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:46.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 0x7f3b2c19cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:46.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c19d3f0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c19d560 con 0x7f3b2c0ffcf0 2023-07-14T18:22:46.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.179+0000 7f3b31da0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c19d720 con 0x7f3b2c100680 2023-07-14T18:22:46.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:46.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46626/0 (socket says 172.21.15.64:46626) 2023-07-14T18:22:46.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 -- 172.21.15.64:0/4052818609 learned_addr learned my addr 172.21.15.64:0/4052818609 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:46.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2bfff700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 0x7f3b2c19cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:46.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2affd700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 0x7f3b2c19a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:46.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 -- 172.21.15.64:0/4052818609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 msgr2=0x7f3b2c19cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:46.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 0x7f3b2c19cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 -- 172.21.15.64:0/4052818609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 msgr2=0x7f3b2c19a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:46.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 0x7f3b2c19a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 -- 172.21.15.64:0/4052818609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c0095f0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b2b7fe700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00b5c0 tx=0x7f3b1c005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:46.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c01d070 con 0x7f3b2c0feb00 2023-07-14T18:22:46.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3b1c003ec0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c19d9a0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1c022a70 con 0x7f3b2c0feb00 2023-07-14T18:22:46.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.180+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c19de20 con 0x7f3b2c0feb00 2023-07-14T18:22:46.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.181+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c1060e0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.182+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3b1c004030 con 0x7f3b2c0feb00 2023-07-14T18:22:46.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.183+0000 7f3b28ff9700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 0x7f3b140a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:46.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.183+0000 7f3b2affd700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 0x7f3b140a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:46.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.183+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3b1c0cdb70 con 0x7f3b2c0feb00 2023-07-14T18:22:46.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.183+0000 7f3b2affd700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 0x7f3b140a36d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005950 tx=0x7f3b2000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:46.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.187+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3b1c0977f0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.432+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b2c02c8e0 con 0x7f3b2c0feb00 2023-07-14T18:22:46.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.433+0000 7f3b28ff9700 1 -- 172.21.15.64:0/4052818609 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4294 (secure 0 0 0) 0x7f3b1c09ae10 con 0x7f3b2c0feb00 2023-07-14T18:22:46.485 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:46.485 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T18:22:46.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 msgr2=0x7f3b140a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:46.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 0x7f3b140a36d0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005950 tx=0x7f3b2000b410 comp rx=0 tx=0).stop 2023-07-14T18:22:46.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 msgr2=0x7f3b2c068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:46.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c00b5c0 tx=0x7f3b1c005740 comp rx=0 tx=0).stop 2023-07-14T18:22:46.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 shutdown_connections 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3b2c100680 0x7f3b2c19cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3b140a1220 0x7f3b140a36d0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b2c0feb00 0x7f3b2c068310 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 --2- 172.21.15.64:0/4052818609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b2c0ffcf0 0x7f3b2c19a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 >> 172.21.15.64:0/4052818609 conn(0x7f3b2c0fa100 msgr2=0x7f3b2c103220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 shutdown_connections 2023-07-14T18:22:46.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:46.435+0000 7f3b31da0700 1 -- 172.21.15.64:0/4052818609 wait complete. 2023-07-14T18:22:46.488 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 6 2023-07-14T18:22:47.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: Deploying daemon mds.c on smithi144 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4052818609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:boot 2023-07-14T18:22:47.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: fsmap 6 up:standby 2023-07-14T18:22:47.184 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:46 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:22:47.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:47.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:47.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:47.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: Deploying daemon mds.c on smithi144 2023-07-14T18:22:47.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4052818609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:47.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:boot 2023-07-14T18:22:47.313 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: fsmap 6 up:standby 2023-07-14T18:22:47.313 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:46 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:22:47.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:47.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:47.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:47.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:47.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: Deploying daemon mds.c on smithi144 2023-07-14T18:22:47.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4052818609' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:47.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:boot 2023-07-14T18:22:47.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: fsmap 6 up:standby 2023-07-14T18:22:47.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:46 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:22:47.797 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:47 smithi144 systemd[1]: Starting Ceph mds.c for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:48.151 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:47 smithi144 podman[124401]: 2023-07-14T18:22:48.152 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:47 smithi144 ceph-mon[98417]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:48.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:47 smithi105 ceph-mon[98013]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:48.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:47 smithi064 ceph-mon[96720]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:48.431 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:48 smithi144 ceph-mds[124440]: -- 172.21.15.144:0/174494036 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d1371eb1e0 con 0x55d1371c0800 2023-07-14T18:22:48.432 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:48 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: starting mds.c at 2023-07-14T18:22:48.432 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:48 smithi144 bash[124401]: b965cece7763dd63fdccc251c548cf80026ace181f66786e1c0edf81e49638dc 2023-07-14T18:22:48.432 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:22:48 smithi144 systemd[1]: Started Ceph mds.c for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:48.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 -- 172.21.15.64:0/4176029868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc100680 msgr2=0x7fc8fc106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 --2- 172.21.15.64:0/4176029868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc100680 0x7fc8fc106f40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc8f400b0a0 tx=0x7fc8f400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 -- 172.21.15.64:0/4176029868 shutdown_connections 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 --2- 172.21.15.64:0/4176029868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc100680 0x7fc8fc106f40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 --2- 172.21.15.64:0/4176029868 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 --2- 172.21.15.64:0/4176029868 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc0ffcf0 0x7fc8fc100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 -- 172.21.15.64:0/4176029868 >> 172.21.15.64:0/4176029868 conn(0x7fc8fc0fa100 msgr2=0x7fc8fc0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 -- 172.21.15.64:0/4176029868 shutdown_connections 2023-07-14T18:22:48.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.588+0000 7fc901d37700 1 -- 172.21.15.64:0/4176029868 wait complete. 2023-07-14T18:22:48.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 Processor -- start 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 -- start start 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0ffcf0 0x7fc8fc194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc100680 0x7fc8fc19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc19c360 con 0x7fc8fc0feb00 2023-07-14T18:22:48.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc19c4a0 con 0x7fc8fc0ffcf0 2023-07-14T18:22:48.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc901d37700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc19c630 con 0x7fc8fc100680 2023-07-14T18:22:48.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc8fb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:48.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc8fb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46640/0 (socket says 172.21.15.64:46640) 2023-07-14T18:22:48.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.589+0000 7fc8fb7fe700 1 -- 172.21.15.64:0/3900983351 learned_addr learned my addr 172.21.15.64:0/3900983351 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8faffd700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0ffcf0 0x7fc8fc194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fbfff700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc100680 0x7fc8fc19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 -- 172.21.15.64:0/3900983351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc100680 msgr2=0x7fc8fc19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc100680 0x7fc8fc19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 -- 172.21.15.64:0/3900983351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0ffcf0 msgr2=0x7fc8fc194d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0ffcf0 0x7fc8fc194d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 -- 172.21.15.64:0/3900983351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f4009d00 con 0x7fc8fc0feb00 2023-07-14T18:22:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8fb7fe700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e970 tx=0x7fc8ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec009790 con 0x7fc8fc0feb00 2023-07-14T18:22:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc8ec00c7e0 con 0x7fc8fc0feb00 2023-07-14T18:22:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc067800 con 0x7fc8fc0feb00 2023-07-14T18:22:48.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec003bd0 con 0x7fc8fc0feb00 2023-07-14T18:22:48.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.590+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc067d60 con 0x7fc8fc0feb00 2023-07-14T18:22:48.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.591+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8fc10a190 con 0x7fc8fc0feb00 2023-07-14T18:22:48.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.592+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc8ec00c950 con 0x7fc8fc0feb00 2023-07-14T18:22:48.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.593+0000 7fc8f8ff9700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 0x7fc8e40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:48.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.593+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8ec014070 con 0x7fc8fc0feb00 2023-07-14T18:22:48.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.593+0000 7fc8faffd700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 0x7fc8e40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:48.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.593+0000 7fc8faffd700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 0x7fc8e40a36d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0005950 tx=0x7fc8f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:48.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.597+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc8ec096170 con 0x7fc8fc0feb00 2023-07-14T18:22:48.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.841+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8fc1059a0 con 0x7fc8fc0feb00 2023-07-14T18:22:48.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.841+0000 7fc8f8ff9700 1 -- 172.21.15.64:0/3900983351 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5054 (secure 0 0 0) 0x7fc8ec016330 con 0x7fc8fc0feb00 2023-07-14T18:22:48.892 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:48.892 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7}],"filesystems":[]} 2023-07-14T18:22:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 msgr2=0x7fc8e40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 0x7fc8e40a36d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0005950 tx=0x7fc8f000b410 comp rx=0 tx=0).stop 2023-07-14T18:22:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 msgr2=0x7fc8fc1947e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec00e970 tx=0x7fc8ec00ec80 comp rx=0 tx=0).stop 2023-07-14T18:22:48.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 shutdown_connections 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc8fc100680 0x7fc8fc19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc8e40a1220 0x7fc8e40a36d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc8fc0feb00 0x7fc8fc1947e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 --2- 172.21.15.64:0/3900983351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8fc0ffcf0 0x7fc8fc194d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 >> 172.21.15.64:0/3900983351 conn(0x7fc8fc0fa100 msgr2=0x7fc8fc0fc2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 shutdown_connections 2023-07-14T18:22:48.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:48.844+0000 7fc901d37700 1 -- 172.21.15.64:0/3900983351 wait complete. 2023-07-14T18:22:48.896 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 7 2023-07-14T18:22:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.559 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: Deploying daemon mds.h on smithi105 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3900983351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:boot 2023-07-14T18:22:49.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: fsmap 7 up:standby 2023-07-14T18:22:49.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:49 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:22:49.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:49.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:49.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:49.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: Deploying daemon mds.h on smithi105 2023-07-14T18:22:49.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3900983351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:49.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:boot 2023-07-14T18:22:49.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: fsmap 7 up:standby 2023-07-14T18:22:49.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:49 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:22:49.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:49.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: Deploying daemon mds.h on smithi105 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3900983351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:boot 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: fsmap 7 up:standby 2023-07-14T18:22:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:49 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:22:49.810 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:49 smithi105 systemd[1]: Starting Ceph mds.h for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:50.383 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:50 smithi105 ceph-mon[98013]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:50.383 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:50 smithi105 podman[122905]: 2023-07-14T18:22:50.383 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:50 smithi105 ceph-mds[122946]: -- 172.21.15.105:0/2104174746 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5556d2e5b1e0 con 0x5556d2e30c00 2023-07-14T18:22:50.383 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:50 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: starting mds.h at 2023-07-14T18:22:50.384 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:50 smithi105 bash[122905]: 093ddda7cff0581393be8fc45304085759a1e5386db1eb8aba62288188772cf6 2023-07-14T18:22:50.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:50 smithi064 ceph-mon[96720]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:50.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:50 smithi144 ceph-mon[98417]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:50.810 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:22:50 smithi105 systemd[1]: Started Ceph mds.h for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:50.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:51.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 -- 172.21.15.64:0/484692751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3481006c0 msgr2=0x7fd348106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:51.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 --2- 172.21.15.64:0/484692751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3481006c0 0x7fd348106f80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd34000b0a0 tx=0x7fd34000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 -- 172.21.15.64:0/484692751 shutdown_connections 2023-07-14T18:22:51.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 --2- 172.21.15.64:0/484692751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 0x7fd3480fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 --2- 172.21.15.64:0/484692751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3481006c0 0x7fd348106f80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 --2- 172.21.15.64:0/484692751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3480ffd30 0x7fd348100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 -- 172.21.15.64:0/484692751 >> 172.21.15.64:0/484692751 conn(0x7fd3480fa160 msgr2=0x7fd3480fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:51.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 -- 172.21.15.64:0/484692751 shutdown_connections 2023-07-14T18:22:51.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.997+0000 7fd34e75e700 1 -- 172.21.15.64:0/484692751 wait complete. 2023-07-14T18:22:51.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 Processor -- start 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 -- start start 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 0x7fd348195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3481006c0 0x7fd34819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34819d4d0 con 0x7fd3480ffd30 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34819d610 con 0x7fd3481006c0 2023-07-14T18:22:51.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34e75e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34819d7a0 con 0x7fd3480feb40 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd3477fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd3477fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46656/0 (socket says 172.21.15.64:46656) 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd3477fe700 1 -- 172.21.15.64:0/1661596947 learned_addr learned my addr 172.21.15.64:0/1661596947 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd347fff700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 0x7fd348195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.998+0000 7fd34ccfb700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3481006c0 0x7fd34819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 -- 172.21.15.64:0/1661596947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 msgr2=0x7fd348195950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:51.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 0x7fd348195950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 -- 172.21.15.64:0/1661596947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3481006c0 msgr2=0x7fd34819cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3481006c0 0x7fd34819cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 -- 172.21.15.64:0/1661596947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340009d00 con 0x7fd3480ffd30 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3477fe700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00d950 tx=0x7fd33c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00f890 con 0x7fd3480ffd30 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd33c00fe80 con 0x7fd3480ffd30 2023-07-14T18:22:51.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34819da80 con 0x7fd3480ffd30 2023-07-14T18:22:51.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33c00e3b0 con 0x7fd3480ffd30 2023-07-14T18:22:51.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:50.999+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34819df90 con 0x7fd3480ffd30 2023-07-14T18:22:51.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.000+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34810a220 con 0x7fd3480ffd30 2023-07-14T18:22:51.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.002+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd33c010460 con 0x7fd3480ffd30 2023-07-14T18:22:51.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.003+0000 7fd3457fa700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 0x7fd3300a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:51.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.003+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd33c0cc1e0 con 0x7fd3480ffd30 2023-07-14T18:22:51.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.003+0000 7fd347fff700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 0x7fd3300a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:51.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.003+0000 7fd347fff700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 0x7fd3300a3650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd3480ffb90 tx=0x7fd338009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:51.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.006+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd33c095dd0 con 0x7fd3480ffd30 2023-07-14T18:22:51.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.252+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd34810a410 con 0x7fd3480ffd30 2023-07-14T18:22:51.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.253+0000 7fd3457fa700 1 -- 172.21.15.64:0/1661596947 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6574 (secure 0 0 0) 0x7fd33c0993f0 con 0x7fd3480ffd30 2023-07-14T18:22:51.303 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T18:22:51.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 msgr2=0x7fd3300a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 0x7fd3300a3650 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd3480ffb90 tx=0x7fd338009260 comp rx=0 tx=0).stop 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 msgr2=0x7fd348195e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd33c00d950 tx=0x7fd33c00dc60 comp rx=0 tx=0).stop 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 shutdown_connections 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd3480feb40 0x7fd348195950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd3300a11a0 0x7fd3300a3650 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd3481006c0 0x7fd34819cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 --2- 172.21.15.64:0/1661596947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd3480ffd30 0x7fd348195e90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 >> 172.21.15.64:0/1661596947 conn(0x7fd3480fa160 msgr2=0x7fd3480fa920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 shutdown_connections 2023-07-14T18:22:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:51.255+0000 7fd34e75e700 1 -- 172.21.15.64:0/1661596947 wait complete. 2023-07-14T18:22:51.307 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 9 2023-07-14T18:22:51.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: Deploying daemon mds.i on smithi144 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:boot 2023-07-14T18:22:51.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: fsmap 8 up:standby 2023-07-14T18:22:51.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:22:51.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:51 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1661596947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:51.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: Deploying daemon mds.i on smithi144 2023-07-14T18:22:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:boot 2023-07-14T18:22:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: fsmap 8 up:standby 2023-07-14T18:22:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:22:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:51 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1661596947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:51.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:51.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: Deploying daemon mds.i on smithi144 2023-07-14T18:22:51.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:boot 2023-07-14T18:22:51.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: fsmap 8 up:standby 2023-07-14T18:22:51.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:22:51.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1661596947' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:52.182 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 systemd[1]: Starting Ceph mds.i for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:52.644 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:52 smithi144 ceph-mon[98417]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:52.644 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 podman[124741]: 2023-07-14T18:22:52.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:52 smithi105 ceph-mon[98013]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:52.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:52 smithi064 ceph-mon[96720]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:52.932 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 ceph-mds[124781]: -- 172.21.15.144:0/1705770933 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aaf5e771e0 con 0x55aaf5e4d000 2023-07-14T18:22:52.933 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 bash[124741]: 20d27cab515dc586a6db9df3654adb8090d089426c766835c24969d1575b20fe 2023-07-14T18:22:52.933 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: starting mds.i at 2023-07-14T18:22:52.933 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:22:52 smithi144 systemd[1]: Started Ceph mds.i for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:53.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:53.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- 172.21.15.64:0/525939741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b4105910 msgr2=0x7f02b4107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 --2- 172.21.15.64:0/525939741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b4105910 0x7f02b4107d70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00b0a0 tx=0x7f02ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- 172.21.15.64:0/525939741 shutdown_connections 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 --2- 172.21.15.64:0/525939741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b4105910 0x7f02b4107d70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 --2- 172.21.15.64:0/525939741 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b41006d0 0x7f02b4102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 --2- 172.21.15.64:0/525939741 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4102ff0 0x7f02b41053d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- 172.21.15.64:0/525939741 >> 172.21.15.64:0/525939741 conn(0x7f02b40fa150 msgr2=0x7f02b40fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:53.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- 172.21.15.64:0/525939741 shutdown_connections 2023-07-14T18:22:53.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- 172.21.15.64:0/525939741 wait complete. 2023-07-14T18:22:53.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 Processor -- start 2023-07-14T18:22:53.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.407+0000 7f02b98a7700 1 -- start start 2023-07-14T18:22:53.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:53.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b4102ff0 0x7f02b4195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:53.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4105910 0x7f02b419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:53.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b419d4a0 con 0x7f02b41006d0 2023-07-14T18:22:53.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b419d5e0 con 0x7f02b4102ff0 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b419d770 con 0x7f02b4105910 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46668/0 (socket says 172.21.15.64:46668) 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 -- 172.21.15.64:0/1402370027 learned_addr learned my addr 172.21.15.64:0/1402370027 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b27fc700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b4102ff0 0x7f02b4195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b37fe700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4105910 0x7f02b419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:53.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 -- 172.21.15.64:0/1402370027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4105910 msgr2=0x7f02b419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4105910 0x7f02b419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 -- 172.21.15.64:0/1402370027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b4102ff0 msgr2=0x7f02b4195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b4102ff0 0x7f02b4195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 -- 172.21.15.64:0/1402370027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac009d00 con 0x7f02b41006d0 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b2ffd700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c9b0 tx=0x7f02a400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4007b00 con 0x7f02b41006d0 2023-07-14T18:22:53.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f02a4007c60 con 0x7f02b41006d0 2023-07-14T18:22:53.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b419da50 con 0x7f02b41006d0 2023-07-14T18:22:53.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4018660 con 0x7f02b41006d0 2023-07-14T18:22:53.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.408+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b419dfb0 con 0x7f02b41006d0 2023-07-14T18:22:53.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.409+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b40fbcc0 con 0x7f02b41006d0 2023-07-14T18:22:53.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.411+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f02a400f450 con 0x7f02b41006d0 2023-07-14T18:22:53.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.411+0000 7f02b88a5700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 0x7f029c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:53.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.411+0000 7f02b27fc700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 0x7f029c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:53.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.412+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f02a40cc480 con 0x7f02b41006d0 2023-07-14T18:22:53.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.412+0000 7f02b27fc700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 0x7f029c0a36d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02a8005950 tx=0x7f02a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:53.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.416+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02a4096070 con 0x7f02b41006d0 2023-07-14T18:22:53.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.673+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02b402c8e0 con 0x7f02b41006d0 2023-07-14T18:22:53.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.673+0000 7f02b88a5700 1 -- 172.21.15.64:0/1402370027 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7336 (secure 0 0 0) 0x7f02a4099690 con 0x7f02b41006d0 2023-07-14T18:22:53.724 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:53.724 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 msgr2=0x7f029c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 0x7f029c0a36d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f02a8005950 tx=0x7f02a80058e0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 msgr2=0x7f02b4195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c9b0 tx=0x7f02a400cd70 comp rx=0 tx=0).stop 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 shutdown_connections 2023-07-14T18:22:53.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02b4105910 0x7f02b419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f029c0a1220 0x7f029c0a36d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02b41006d0 0x7f02b4195920 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 --2- 172.21.15.64:0/1402370027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02b4102ff0 0x7f02b4195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 >> 172.21.15.64:0/1402370027 conn(0x7f02b40fa150 msgr2=0x7f02b4101330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 shutdown_connections 2023-07-14T18:22:53.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:53.675+0000 7f02b98a7700 1 -- 172.21.15.64:0/1402370027 wait complete. 2023-07-14T18:22:53.728 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 10 2023-07-14T18:22:53.907 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:53.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:53.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:53.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:53.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:53.908 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:53.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: Deploying daemon mds.g on smithi064 2023-07-14T18:22:53.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:boot 2023-07-14T18:22:53.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: fsmap 9 up:standby 2023-07-14T18:22:53.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:22:53.909 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:53 smithi064 ceph-mon[96720]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:54.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:54.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: Deploying daemon mds.g on smithi064 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:boot 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: fsmap 9 up:standby 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:22:54.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:53 smithi105 ceph-mon[98013]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:54.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:54.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:54.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:54.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:54.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: Deploying daemon mds.g on smithi064 2023-07-14T18:22:54.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:boot 2023-07-14T18:22:54.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: fsmap 9 up:standby 2023-07-14T18:22:54.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:22:54.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:53 smithi144 ceph-mon[98417]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:54.591 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 systemd[1]: Starting Ceph mds.g for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:54.912 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:54 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1402370027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:54.913 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 podman[131062]: 2023-07-14T18:22:54.913 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 ceph-mds[131102]: -- 172.21.15.64:0/1654520247 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557da94251e0 con 0x557da93fac00 2023-07-14T18:22:54.913 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: starting mds.g at 2023-07-14T18:22:55.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:54 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1402370027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:55.181 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:54 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1402370027' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:55.341 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 bash[131062]: c8fb439b7b058b5b5b44ba38e680a218c2ffbbea5caf1106c5eb26b163024b86 2023-07-14T18:22:55.341 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:22:54 smithi064 systemd[1]: Started Ceph mds.g for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:55.739 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:55.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- 172.21.15.64:0/3730954991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2314100610 msgr2=0x7f2314106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:55.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 --2- 172.21.15.64:0/3730954991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2314100610 0x7f2314106ed0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f230c00b0a0 tx=0x7f230c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- 172.21.15.64:0/3730954991 shutdown_connections 2023-07-14T18:22:55.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 --2- 172.21.15.64:0/3730954991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2314100610 0x7f2314106ed0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 --2- 172.21.15.64:0/3730954991 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23140fea90 0x7f23140feea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 --2- 172.21.15.64:0/3730954991 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140ffc80 0x7f23141000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- 172.21.15.64:0/3730954991 >> 172.21.15.64:0/3730954991 conn(0x7f23140fa090 msgr2=0x7f23140fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:55.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- 172.21.15.64:0/3730954991 shutdown_connections 2023-07-14T18:22:55.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- 172.21.15.64:0/3730954991 wait complete. 2023-07-14T18:22:55.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 Processor -- start 2023-07-14T18:22:55.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.833+0000 7f231b7a9700 1 -- start start 2023-07-14T18:22:55.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 0x7f23141958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:55.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 0x7f231419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231419d450 con 0x7f23140ffc80 2023-07-14T18:22:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231419d590 con 0x7f2314100610 2023-07-14T18:22:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f231419d720 con 0x7f23140fea90 2023-07-14T18:22:55.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46698/0 (socket says 172.21.15.64:46698) 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 -- 172.21.15.64:0/433956591 learned_addr learned my addr 172.21.15.64:0/433956591 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2319d46700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 0x7f231419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2319545700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 0x7f23141958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 -- 172.21.15.64:0/433956591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 msgr2=0x7f23141958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:55.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 0x7f23141958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 -- 172.21.15.64:0/433956591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 msgr2=0x7f231419ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:55.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 0x7f231419ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:55.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 -- 172.21.15.64:0/433956591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c009d00 con 0x7f23140ffc80 2023-07-14T18:22:55.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2319d46700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 0x7f231419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:55.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2319545700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 0x7f23141958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f2318d44700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f230400d980 tx=0x7f230400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f230400bb60 con 0x7f23140ffc80 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2304007640 con 0x7f23140ffc80 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f231419da00 con 0x7f23140ffc80 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2304004d60 con 0x7f23140ffc80 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.834+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f231419df10 con 0x7f23140ffc80 2023-07-14T18:22:55.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.835+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2314106b40 con 0x7f23140ffc80 2023-07-14T18:22:55.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.837+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2304004020 con 0x7f23140ffc80 2023-07-14T18:22:55.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.837+0000 7f230a7fc700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 0x7f23000a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:55.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.838+0000 7f2319545700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 0x7f23000a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:55.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.838+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f23040cc850 con 0x7f23140ffc80 2023-07-14T18:22:55.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.838+0000 7f2319545700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 0x7f23000a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f23140ffae0 tx=0x7f2310009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:55.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:55.841+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2304096440 con 0x7f23140ffc80 2023-07-14T18:22:56.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.093+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f231402c8e0 con 0x7f23140ffc80 2023-07-14T18:22:56.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.094+0000 7f230a7fc700 1 -- 172.21.15.64:0/433956591 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8094 (secure 0 0 0) 0x7f230400de60 con 0x7f23140ffc80 2023-07-14T18:22:56.145 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:56.146 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:22:56.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 msgr2=0x7f23000a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:56.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 0x7f23000a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f23140ffae0 tx=0x7f2310009260 comp rx=0 tx=0).stop 2023-07-14T18:22:56.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 msgr2=0x7f2314195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:56.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f230400d980 tx=0x7f230400dd40 comp rx=0 tx=0).stop 2023-07-14T18:22:56.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 shutdown_connections 2023-07-14T18:22:56.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f23140fea90 0x7f23141958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:56.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23000a11a0 0x7f23000a3650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:56.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2314100610 0x7f231419ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:56.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 --2- 172.21.15.64:0/433956591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f23140ffc80 0x7f2314195e10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:56.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.096+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 >> 172.21.15.64:0/433956591 conn(0x7f23140fa090 msgr2=0x7f23140fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:56.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.097+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 shutdown_connections 2023-07-14T18:22:56.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:56.097+0000 7f231b7a9700 1 -- 172.21.15.64:0/433956591 wait complete. 2023-07-14T18:22:56.152 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 11 2023-07-14T18:22:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: Deploying daemon mds.a on smithi064 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6838/3820392799,v1:172.21.15.64:6839/3820392799] up:boot 2023-07-14T18:22:56.312 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: fsmap 10 up:standby 2023-07-14T18:22:56.313 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:55 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:22:56.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: Deploying daemon mds.a on smithi064 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:56.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6838/3820392799,v1:172.21.15.64:6839/3820392799] up:boot 2023-07-14T18:22:56.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: fsmap 10 up:standby 2023-07-14T18:22:56.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:55 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:22:56.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:56.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:56.432 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: Deploying daemon mds.a on smithi064 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6838/3820392799,v1:172.21.15.64:6839/3820392799] up:boot 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: fsmap 10 up:standby 2023-07-14T18:22:56.433 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:55 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:22:56.945 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:56 smithi064 systemd[1]: Starting Ceph mds.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:57.228 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:56 smithi064 podman[131447]: 2023-07-14T18:22:57.229 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:56 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/433956591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:57.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:56 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/433956591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:57.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:56 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/433956591' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:57.591 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:57 smithi064 bash[131447]: a11e7f51b664d0304efad0a0653ed1c1e01427f4560e68ae22fe4c639f80e1a8 2023-07-14T18:22:57.591 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:57 smithi064 ceph-mds[131487]: -- 172.21.15.64:0/4031997780 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564a741371e0 con 0x564a7410cc00 2023-07-14T18:22:57.592 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: starting mds.a at 2023-07-14T18:22:57.592 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:22:57 smithi064 systemd[1]: Started Ceph mds.a for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:22:58.160 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:22:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 -- 172.21.15.64:0/1759263945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 msgr2=0x7f9d60100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/1759263945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60100140 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009990 tx=0x7f9d5c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 -- 172.21.15.64:0/1759263945 shutdown_connections 2023-07-14T18:22:58.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/1759263945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d600fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/1759263945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 0x7f9d60106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/1759263945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60100140 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.256+0000 7f9d67a9e700 1 -- 172.21.15.64:0/1759263945 >> 172.21.15.64:0/1759263945 conn(0x7f9d600fa100 msgr2=0x7f9d600fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- 172.21.15.64:0/1759263945 shutdown_connections 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- 172.21.15.64:0/1759263945 wait complete. 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 Processor -- start 2023-07-14T18:22:58.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- start start 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d60194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 0x7f9d6019be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6019c410 con 0x7f9d600ffcf0 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6019c550 con 0x7f9d60100680 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d67a9e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6019c6e0 con 0x7f9d600feb00 2023-07-14T18:22:58.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d65039700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d65039700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46726/0 (socket says 172.21.15.64:46726) 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.257+0000 7f9d65039700 1 -- 172.21.15.64:0/3647700619 learned_addr learned my addr 172.21.15.64:0/3647700619 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d6583a700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d60194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d6603b700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 0x7f9d6019be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 -- 172.21.15.64:0/3647700619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 msgr2=0x7f9d60194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:58.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d60194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 -- 172.21.15.64:0/3647700619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 msgr2=0x7f9d6019be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 0x7f9d6019be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 -- 172.21.15.64:0/3647700619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c0095f0 con 0x7f9d600ffcf0 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d6583a700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d60194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d65039700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006010 tx=0x7f9d5c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c01d070 con 0x7f9d600ffcf0 2023-07-14T18:22:58.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9d5c004100 con 0x7f9d600ffcf0 2023-07-14T18:22:58.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d60067860 con 0x7f9d600ffcf0 2023-07-14T18:22:58.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c005290 con 0x7f9d600ffcf0 2023-07-14T18:22:58.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.258+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d60067d10 con 0x7f9d600ffcf0 2023-07-14T18:22:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.259+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6010a270 con 0x7f9d600ffcf0 2023-07-14T18:22:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.260+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9d5c0047e0 con 0x7f9d600ffcf0 2023-07-14T18:22:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.261+0000 7f9d52ffd700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 0x7f9d4c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:22:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.261+0000 7f9d6583a700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 0x7f9d4c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:22:58.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.261+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9d5c0cd640 con 0x7f9d600ffcf0 2023-07-14T18:22:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.262+0000 7f9d6583a700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 0x7f9d4c0a3650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d600ffb50 tx=0x7f9d54008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:22:58.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.266+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d5c097220 con 0x7f9d600ffcf0 2023-07-14T18:22:58.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: Deploying daemon mds.b on smithi105 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:58.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6840/146209694,v1:172.21.15.64:6841/146209694] up:boot 2023-07-14T18:22:58.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: fsmap 11 up:standby 2023-07-14T18:22:58.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:58 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:22:58.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.513+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d6010a430 con 0x7f9d600ffcf0 2023-07-14T18:22:58.564 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.514+0000 7f9d52ffd700 1 -- 172.21.15.64:0/3647700619 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8849 (secure 0 0 0) 0x7f9d5c09a840 con 0x7f9d600ffcf0 2023-07-14T18:22:58.564 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:22:58.564 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T18:22:58.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 msgr2=0x7f9d4c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:58.567 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 0x7f9d4c0a3650 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d600ffb50 tx=0x7f9d54008040 comp rx=0 tx=0).stop 2023-07-14T18:22:58.567 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 msgr2=0x7f9d60194dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:22:58.567 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c006010 tx=0x7f9d5c005ef0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.567 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 shutdown_connections 2023-07-14T18:22:58.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9d600feb00 0x7f9d60194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9d4c0a11a0 0x7f9d4c0a3650 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9d60100680 0x7f9d6019be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 --2- 172.21.15.64:0/3647700619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d600ffcf0 0x7f9d60194dd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:22:58.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 >> 172.21.15.64:0/3647700619 conn(0x7f9d600fa100 msgr2=0x7f9d60103280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:22:58.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 shutdown_connections 2023-07-14T18:22:58.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:22:58.516+0000 7f9d67a9e700 1 -- 172.21.15.64:0/3647700619 wait complete. 2023-07-14T18:22:58.569 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 12 2023-07-14T18:22:58.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:58.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: Deploying daemon mds.b on smithi105 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6840/146209694,v1:172.21.15.64:6841/146209694] up:boot 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: fsmap 11 up:standby 2023-07-14T18:22:58.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:58 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:22:58.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: Deploying daemon mds.b on smithi105 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:22:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6840/146209694,v1:172.21.15.64:6841/146209694] up:boot 2023-07-14T18:22:58.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: fsmap 11 up:standby 2023-07-14T18:22:58.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:58 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:22:59.060 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:58 smithi105 systemd[1]: Starting Ceph mds.b for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:22:59.412 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:22:59 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3647700619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:59.413 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:59 smithi105 podman[123248]: 2023-07-14T18:22:59.413 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:59 smithi105 ceph-mds[123288]: -- 172.21.15.105:0/2990685433 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1006071e0 con 0x55f1005dcc00 2023-07-14T18:22:59.413 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:59 smithi105 bash[123248]: 7f3b040aee5921e518fb0b851b4a02f03a98fbf5e29cb7233d2791b6bd5d2dc3 2023-07-14T18:22:59.413 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:59 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: starting mds.b at 2023-07-14T18:22:59.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:22:59 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3647700619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:59.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:22:59 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3647700619' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:22:59.810 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:22:59 smithi105 systemd[1]: Started Ceph mds.b for 5441c8fe-2272-11ee-9b33-001a4aab830c. 2023-07-14T18:23:00.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:23:00.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 -- 172.21.15.64:0/1078920927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 msgr2=0x7f414c106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 --2- 172.21.15.64:0/1078920927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c106f40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f414400b0a0 tx=0x7f414400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 -- 172.21.15.64:0/1078920927 shutdown_connections 2023-07-14T18:23:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 --2- 172.21.15.64:0/1078920927 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0feb00 0x7f414c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 --2- 172.21.15.64:0/1078920927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c106f40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 --2- 172.21.15.64:0/1078920927 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0ffcf0 0x7f414c100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 -- 172.21.15.64:0/1078920927 >> 172.21.15.64:0/1078920927 conn(0x7f414c0fa100 msgr2=0x7f414c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:00.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 -- 172.21.15.64:0/1078920927 shutdown_connections 2023-07-14T18:23:00.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 -- 172.21.15.64:0/1078920927 wait complete. 2023-07-14T18:23:00.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.664+0000 7f4151ea2700 1 Processor -- start 2023-07-14T18:23:00.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 -- start start 2023-07-14T18:23:00.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0feb00 0x7f414c195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0ffcf0 0x7f414c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c19d4b0 con 0x7f414c100680 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c19d5f0 con 0x7f414c0feb00 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f4151ea2700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414c19d780 con 0x7f414c0ffcf0 2023-07-14T18:23:00.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46740/0 (socket says 172.21.15.64:46740) 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 -- 172.21.15.64:0/3010735490 learned_addr learned my addr 172.21.15.64:0/3010735490 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414b7fe700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0feb00 0x7f414c195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414affd700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0ffcf0 0x7f414c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 -- 172.21.15.64:0/3010735490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0ffcf0 msgr2=0x7f414c195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0ffcf0 0x7f414c195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 -- 172.21.15.64:0/3010735490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0feb00 msgr2=0x7f414c195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0feb00 0x7f414c195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 -- 172.21.15.64:0/3010735490 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4144009d00 con 0x7f414c100680 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.665+0000 7f414bfff700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f414400bad0 tx=0x7f414400bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.666+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414400e040 con 0x7f414c100680 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.666+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4144004e40 con 0x7f414c100680 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.666+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41440052a0 con 0x7f414c100680 2023-07-14T18:23:00.721 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.666+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f414c19da00 con 0x7f414c100680 2023-07-14T18:23:00.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.666+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f414c19de80 con 0x7f414c100680 2023-07-14T18:23:00.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.667+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414c106b90 con 0x7f414c100680 2023-07-14T18:23:00.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.668+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f414401d990 con 0x7f414c100680 2023-07-14T18:23:00.722 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.668+0000 7f4148ff9700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 0x7f41340a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:00.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.668+0000 7f414b7fe700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 0x7f41340a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:00.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.668+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41440cda90 con 0x7f414c100680 2023-07-14T18:23:00.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.669+0000 7f414b7fe700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 0x7f41340a36d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ffb50 tx=0x7f413c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:00.723 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.673+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4144097710 con 0x7f414c100680 2023-07-14T18:23:00.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:boot 2023-07-14T18:23:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: fsmap 12 up:standby 2023-07-14T18:23:00.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:00 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:23:00.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:boot 2023-07-14T18:23:00.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: fsmap 12 up:standby 2023-07-14T18:23:00.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:00 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:23:00.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:boot 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: fsmap 12 up:standby 2023-07-14T18:23:00.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:00 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:23:00.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.925+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f414c02c8e0 con 0x7f414c100680 2023-07-14T18:23:00.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.926+0000 7f4148ff9700 1 -- 172.21.15.64:0/3010735490 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9610 (secure 0 0 0) 0x7f414409ad30 con 0x7f414c100680 2023-07-14T18:23:00.977 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:00.977 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T18:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 msgr2=0x7f41340a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 0x7f41340a36d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f414c0ffb50 tx=0x7f413c00b410 comp rx=0 tx=0).stop 2023-07-14T18:23:00.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 msgr2=0x7f414c19cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f414400bad0 tx=0x7f414400bb00 comp rx=0 tx=0).stop 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 shutdown_connections 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41340a1220 0x7f41340a36d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f414c100680 0x7f414c19cee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f414c0feb00 0x7f414c195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 --2- 172.21.15.64:0/3010735490 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f414c0ffcf0 0x7f414c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 >> 172.21.15.64:0/3010735490 conn(0x7f414c0fa100 msgr2=0x7f414c0fc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 shutdown_connections 2023-07-14T18:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:00.929+0000 7f4151ea2700 1 -- 172.21.15.64:0/3010735490 wait complete. 2023-07-14T18:23:00.981 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 13 2023-07-14T18:23:00.994 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:01.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 -- 172.21.15.64:0/2891875263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c1018e0 msgr2=0x7ff85c1081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 --2- 172.21.15.64:0/2891875263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c1018e0 0x7ff85c1081a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff85400b0a0 tx=0x7ff85400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 -- 172.21.15.64:0/2891875263 shutdown_connections 2023-07-14T18:23:01.123 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 --2- 172.21.15.64:0/2891875263 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c0fdee0 0x7ff85c0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 --2- 172.21.15.64:0/2891875263 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c1018e0 0x7ff85c1081a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 --2- 172.21.15.64:0/2891875263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c100f50 0x7ff85c1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 -- 172.21.15.64:0/2891875263 >> 172.21.15.64:0/2891875263 conn(0x7ff85c0f98a0 msgr2=0x7ff85c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:01.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 -- 172.21.15.64:0/2891875263 shutdown_connections 2023-07-14T18:23:01.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.072+0000 7ff860d85700 1 -- 172.21.15.64:0/2891875263 wait complete. 2023-07-14T18:23:01.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 Processor -- start 2023-07-14T18:23:01.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 -- start start 2023-07-14T18:23:01.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c0fdee0 0x7ff85c1008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c100f50 0x7ff85c0fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0ffbe0 con 0x7ff85c1018e0 2023-07-14T18:23:01.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c0ffd50 con 0x7ff85c0fdee0 2023-07-14T18:23:01.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff860d85700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85c19f970 con 0x7ff85c100f50 2023-07-14T18:23:01.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff85ad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff85ad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46754/0 (socket says 172.21.15.64:46754) 2023-07-14T18:23:01.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.073+0000 7ff85ad9d700 1 -- 172.21.15.64:0/2820353675 learned_addr learned my addr 172.21.15.64:0/2820353675 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:01.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff859d9b700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c100f50 0x7ff85c0fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 -- 172.21.15.64:0/2820353675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c100f50 msgr2=0x7ff85c0fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c100f50 0x7ff85c0fef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 -- 172.21.15.64:0/2820353675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c0fdee0 msgr2=0x7ff85c1008c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c0fdee0 0x7ff85c1008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 -- 172.21.15.64:0/2820353675 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff854009d00 con 0x7ff85c1018e0 2023-07-14T18:23:01.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff85ad9d700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff85400b780 tx=0x7ff8540078e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85400e040 con 0x7ff85c1018e0 2023-07-14T18:23:01.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85c19fbf0 con 0x7ff85c1018e0 2023-07-14T18:23:01.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff854004c50 con 0x7ff85c1018e0 2023-07-14T18:23:01.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff854005260 con 0x7ff85c1018e0 2023-07-14T18:23:01.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.074+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85c1a0070 con 0x7ff85c1018e0 2023-07-14T18:23:01.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.075+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85c10b230 con 0x7ff85c1018e0 2023-07-14T18:23:01.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.076+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff85401d470 con 0x7ff85c1018e0 2023-07-14T18:23:01.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.077+0000 7ff8537fe700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 0x7ff8480a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.077+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8540cdd80 con 0x7ff85c1018e0 2023-07-14T18:23:01.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.077+0000 7ff85a59c700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 0x7ff8480a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.077+0000 7ff85a59c700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 0x7ff8480a3650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0674e0 tx=0x7ff84c0096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.080+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff854097a00 con 0x7ff85c1018e0 2023-07-14T18:23:01.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.290+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85c106ca0 con 0x7ff85c1018e0 2023-07-14T18:23:01.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.292+0000 7ff8537fe700 1 -- 172.21.15.64:0/2820353675 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18409 (secure 0 0 0) 0x7ff85401d720 con 0x7ff85c1018e0 2023-07-14T18:23:01.343 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:01.343 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":77,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:22:24.216131+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:01.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 msgr2=0x7ff8480a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 0x7ff8480a3650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff85c0674e0 tx=0x7ff84c0096c0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 msgr2=0x7ff85c0ff440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff85400b780 tx=0x7ff8540078e0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 shutdown_connections 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff8480a11a0 0x7ff8480a3650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff85c1018e0 0x7ff85c0ff440 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff85c0fdee0 0x7ff85c1008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 --2- 172.21.15.64:0/2820353675 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff85c100f50 0x7ff85c0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 >> 172.21.15.64:0/2820353675 conn(0x7ff85c0f98a0 msgr2=0x7ff85c0fbb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:01.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 shutdown_connections 2023-07-14T18:23:01.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.294+0000 7ff860d85700 1 -- 172.21.15.64:0/2820353675 wait complete. 2023-07-14T18:23:01.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T18:23:01.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:23:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 -- 172.21.15.64:0/2660309394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc100e70 msgr2=0x7f60cc1012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 --2- 172.21.15.64:0/2660309394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc100e70 0x7f60cc1012c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f60c0009990 tx=0x7f60c0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 -- 172.21.15.64:0/2660309394 shutdown_connections 2023-07-14T18:23:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 --2- 172.21.15.64:0/2660309394 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc101800 0x7f60cc1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 --2- 172.21.15.64:0/2660309394 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 0x7f60cc100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 --2- 172.21.15.64:0/2660309394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc100e70 0x7f60cc1012c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 -- 172.21.15.64:0/2660309394 >> 172.21.15.64:0/2660309394 conn(0x7f60cc0fb2a0 msgr2=0x7f60cc0fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:01.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 -- 172.21.15.64:0/2660309394 shutdown_connections 2023-07-14T18:23:01.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.442+0000 7f60d11b9700 1 -- 172.21.15.64:0/2660309394 wait complete. 2023-07-14T18:23:01.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.443+0000 7f60d11b9700 1 Processor -- start 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.443+0000 7f60d11b9700 1 -- start start 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 0x7f60cc0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc100e70 0x7f60cc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc073710 con 0x7f60cc101800 2023-07-14T18:23:01.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc073880 con 0x7f60cc0ffc80 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60d11b9700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc19f8e0 con 0x7f60cc100e70 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46768/0 (socket says 172.21.15.64:46768) 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 -- 172.21.15.64:0/1556023120 learned_addr learned my addr 172.21.15.64:0/1556023120 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cad9d700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 0x7f60cc0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60ca59c700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc100e70 0x7f60cc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 -- 172.21.15.64:0/1556023120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc100e70 msgr2=0x7f60cc072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc100e70 0x7f60cc072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 -- 172.21.15.64:0/1556023120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 msgr2=0x7f60cc0743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 0x7f60cc0743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 -- 172.21.15.64:0/1556023120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c00095f0 con 0x7f60cc101800 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.444+0000 7f60cb59e700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f60c400ef30 tx=0x7f60c400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.445+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4018070 con 0x7f60cc101800 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.445+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f60c400f040 con 0x7f60cc101800 2023-07-14T18:23:01.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.445+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c400cbc0 con 0x7f60cc101800 2023-07-14T18:23:01.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.445+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60cc19fb60 con 0x7f60cc101800 2023-07-14T18:23:01.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.445+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc1a0020 con 0x7f60cc101800 2023-07-14T18:23:01.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.446+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc106c60 con 0x7f60cc101800 2023-07-14T18:23:01.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.448+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f60c400cd80 con 0x7f60cc101800 2023-07-14T18:23:01.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.448+0000 7f60b3fff700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 0x7f60b40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.448+0000 7f60cad9d700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 0x7f60b40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.448+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60c40cc450 con 0x7f60cc101800 2023-07-14T18:23:01.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.449+0000 7f60cad9d700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 0x7f60b40a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f60cc100cd0 tx=0x7f60bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.452+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f60c40960d0 con 0x7f60cc101800 2023-07-14T18:23:01.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.661+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60cc02cb50 con 0x7f60cc101800 2023-07-14T18:23:01.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.661+0000 7f60b3fff700 1 -- 172.21.15.64:0/1556023120 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60c40996f0 con 0x7f60cc101800 2023-07-14T18:23:01.715 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:23:01.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 msgr2=0x7f60b40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 0x7f60b40a3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f60cc100cd0 tx=0x7f60bc009260 comp rx=0 tx=0).stop 2023-07-14T18:23:01.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 msgr2=0x7f60cc072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f60c400ef30 tx=0x7f60c400c3a0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 shutdown_connections 2023-07-14T18:23:01.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f60b40a11d0 0x7f60b40a3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60cc101800 0x7f60cc072f70 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60cc0ffc80 0x7f60cc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 --2- 172.21.15.64:0/1556023120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f60cc100e70 0x7f60cc072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 >> 172.21.15.64:0/1556023120 conn(0x7f60cc0fb2a0 msgr2=0x7f60cc0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:01.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 shutdown_connections 2023-07-14T18:23:01.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.663+0000 7f60d11b9700 1 -- 172.21.15.64:0/1556023120 wait complete. 2023-07-14T18:23:01.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:01 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:01.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3010735490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2820353675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.725 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:01 smithi064 ceph-mon[96720]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:01.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T18:23:01.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:01 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:01.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3010735490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2820353675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:01 smithi105 ceph-mon[98013]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:01.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 -- 172.21.15.64:0/467979915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 msgr2=0x7f81f0100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 --2- 172.21.15.64:0/467979915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0100040 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f81e4009990 tx=0x7f81e4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 -- 172.21.15.64:0/467979915 shutdown_connections 2023-07-14T18:23:01.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 --2- 172.21.15.64:0/467979915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f01017b0 0x7f81f0108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 --2- 172.21.15.64:0/467979915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0100040 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 --2- 172.21.15.64:0/467979915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f0100e20 0x7f81f0101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 -- 172.21.15.64:0/467979915 >> 172.21.15.64:0/467979915 conn(0x7f81f00fb210 msgr2=0x7f81f00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:01.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 -- 172.21.15.64:0/467979915 shutdown_connections 2023-07-14T18:23:01.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.807+0000 7f81f7e25700 1 -- 172.21.15.64:0/467979915 wait complete. 2023-07-14T18:23:01.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 Processor -- start 2023-07-14T18:23:01.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 -- start start 2023-07-14T18:23:01.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f0100e20 0x7f81f0195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f01017b0 0x7f81f019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f019d400 con 0x7f81f00ffc30 2023-07-14T18:23:01.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f019d540 con 0x7f81f0100e20 2023-07-14T18:23:01.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f7e25700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f019d6d0 con 0x7f81f01017b0 2023-07-14T18:23:01.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f5bc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f5bc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46780/0 (socket says 172.21.15.64:46780) 2023-07-14T18:23:01.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f5bc1700 1 -- 172.21.15.64:0/3503641221 learned_addr learned my addr 172.21.15.64:0/3503641221 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:01.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f53c0700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f0100e20 0x7f81f0195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.808+0000 7f81f63c2700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f01017b0 0x7f81f019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 -- 172.21.15.64:0/3503641221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f01017b0 msgr2=0x7f81f019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f01017b0 0x7f81f019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 -- 172.21.15.64:0/3503641221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f0100e20 msgr2=0x7f81f0195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f0100e20 0x7f81f0195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 -- 172.21.15.64:0/3503641221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e40095f0 con 0x7f81f00ffc30 2023-07-14T18:23:01.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f5bc1700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81e40098e0 tx=0x7f81e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e401d070 con 0x7f81f00ffc30 2023-07-14T18:23:01.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f81e400bdd0 con 0x7f81f00ffc30 2023-07-14T18:23:01.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f019d950 con 0x7f81f00ffc30 2023-07-14T18:23:01.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e4022d20 con 0x7f81f00ffc30 2023-07-14T18:23:01.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.809+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f019ddd0 con 0x7f81f00ffc30 2023-07-14T18:23:01.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.810+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f0106b60 con 0x7f81f00ffc30 2023-07-14T18:23:01.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.811+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f81e400b940 con 0x7f81f00ffc30 2023-07-14T18:23:01.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.811+0000 7f81e2ffd700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 0x7f81dc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:01.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.812+0000 7f81f53c0700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 0x7f81dc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:01.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.812+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81e40cda40 con 0x7f81f00ffc30 2023-07-14T18:23:01.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.812+0000 7f81f53c0700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 0x7f81dc0a36d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81ec007c60 tx=0x7f81ec0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:01.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:01.816+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f81e40cfe60 con 0x7f81f00ffc30 2023-07-14T18:23:01.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:01 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:01.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3010735490' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2820353675' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:01.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:01 smithi144 ceph-mon[98417]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:02.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.025+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f81f002cb20 con 0x7f81f00ffc30 2023-07-14T18:23:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.026+0000 7f81e2ffd700 1 -- 172.21.15.64:0/3503641221 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f81e40976c0 con 0x7f81f00ffc30 2023-07-14T18:23:02.077 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:02.077 INFO:teuthology.orchestra.run.smithi064.stdout:[ 2023-07-14T18:23:02.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2023-07-14T18:23:02.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolnum": 1, 2023-07-14T18:23:02.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolname": ".mgr" 2023-07-14T18:23:02.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-07-14T18:23:02.079 INFO:teuthology.orchestra.run.smithi064.stdout:] 2023-07-14T18:23:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 msgr2=0x7f81dc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 0x7f81dc0a36d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81ec007c60 tx=0x7f81ec0073d0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 msgr2=0x7f81f0195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f81e40098e0 tx=0x7f81e4004730 comp rx=0 tx=0).stop 2023-07-14T18:23:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 shutdown_connections 2023-07-14T18:23:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f81f01017b0 0x7f81f019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f81dc0a1220 0x7f81dc0a36d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81f00ffc30 0x7f81f0195880 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 --2- 172.21.15.64:0/3503641221 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f81f0100e20 0x7f81f0195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 >> 172.21.15.64:0/3503641221 conn(0x7f81f00fb210 msgr2=0x7f81f00fd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:02.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 shutdown_connections 2023-07-14T18:23:02.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.028+0000 7f81f7e25700 1 -- 172.21.15.64:0/3503641221 wait complete. 2023-07-14T18:23:02.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T18:23:02.093 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T18:23:02.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1398894784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 msgr2=0x7fcf04106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:02.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1398894784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04106fa0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fcefc00b0a0 tx=0x7fcefc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1398894784 shutdown_connections 2023-07-14T18:23:02.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1398894784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04106fa0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1398894784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1398894784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf040ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1398894784 >> 172.21.15.64:0/1398894784 conn(0x7fcf040f98a0 msgr2=0x7fcf040fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1398894784 shutdown_connections 2023-07-14T18:23:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.176+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1398894784 wait complete. 2023-07-14T18:23:02.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 Processor -- start 2023-07-14T18:23:02.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 -- start start 2023-07-14T18:23:02.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:02.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04101e10 con 0x7fcf040ff6f0 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf04101f80 con 0x7fcf04100180 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf0a6fd700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0419f9b0 con 0x7fcf040fed40 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf03fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf037fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf03fff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:54448/0 (socket says 172.21.15.64:54448) 2023-07-14T18:23:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.177+0000 7fcf03fff700 1 -- 172.21.15.64:0/1437982698 learned_addr learned my addr 172.21.15.64:0/1437982698 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf08c9a700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 -- 172.21.15.64:0/1437982698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 msgr2=0x7fcf04102af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 -- 172.21.15.64:0/1437982698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 msgr2=0x7fcf04101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 -- 172.21.15.64:0/1437982698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef8009520 con 0x7fcf040ff6f0 2023-07-14T18:23:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf08c9a700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:23:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf03fff700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:23:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf037fe700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04101130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcef800e9f0 tx=0x7fcef800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef800cb70 con 0x7fcf040ff6f0 2023-07-14T18:23:02.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcef800ee90 con 0x7fcf040ff6f0 2023-07-14T18:23:02.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef4009520 con 0x7fcf040ff6f0 2023-07-14T18:23:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef8003bc0 con 0x7fcf040ff6f0 2023-07-14T18:23:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.178+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc009d00 con 0x7fcf040ff6f0 2023-07-14T18:23:02.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.179+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0410a180 con 0x7fcf040ff6f0 2023-07-14T18:23:02.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.181+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcef8003d40 con 0x7fcf040ff6f0 2023-07-14T18:23:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.182+0000 7fcf017fa700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.182+0000 7fcf03fff700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.182+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcef8014070 con 0x7fcf040ff6f0 2023-07-14T18:23:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.183+0000 7fcf03fff700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 0x7fceec0a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009fd0 tx=0x7fcef4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:02.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.185+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcef8096440 con 0x7fcf040ff6f0 2023-07-14T18:23:02.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:02.407+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fcf0410a340 con 0x7fcf040ff6f0 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1556023120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3503641221' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.182 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:02 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1556023120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:03.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3503641221' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:23:03.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.311 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:02 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.341 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1556023120' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:03.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3503641221' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T18:23:03.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.342 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.343 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.344 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:02 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:03.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.404+0000 7fcf017fa700 1 -- 172.21.15.64:0/1437982698 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcef8099a60 con 0x7fcf040ff6f0 2023-07-14T18:23:03.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 msgr2=0x7fceec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 0x7fceec0a3600 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcef4009fd0 tx=0x7fcef4009260 comp rx=0 tx=0).stop 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 msgr2=0x7fcf04101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04101130 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fcef800e9f0 tx=0x7fcef800ed00 comp rx=0 tx=0).stop 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 shutdown_connections 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcf040fed40 0x7fcf04102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fceec0a1150 0x7fceec0a3600 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcf04100180 0x7fcf04101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 --2- 172.21.15.64:0/1437982698 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf040ff6f0 0x7fcf04101130 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 >> 172.21.15.64:0/1437982698 conn(0x7fcf040f98a0 msgr2=0x7fcf040fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 shutdown_connections 2023-07-14T18:23:03.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.408+0000 7fcf0a6fd700 1 -- 172.21.15.64:0/1437982698 wait complete. 2023-07-14T18:23:03.461 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' created 2023-07-14T18:23:03.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-14T18:23:03.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 -- 172.21.15.64:0/1803471799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354100e00 msgr2=0x7fe354101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:03.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/1803471799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354100e00 0x7fe354101250 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe348009990 tx=0x7fe348009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 -- 172.21.15.64:0/1803471799 shutdown_connections 2023-07-14T18:23:03.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/1803471799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 0x7fe354100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/1803471799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354101790 0x7fe354108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/1803471799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354100e00 0x7fe354101250 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 -- 172.21.15.64:0/1803471799 >> 172.21.15.64:0/1803471799 conn(0x7fe3540fb210 msgr2=0x7fe3540fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 -- 172.21.15.64:0/1803471799 shutdown_connections 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.560+0000 7fe35c6bb700 1 -- 172.21.15.64:0/1803471799 wait complete. 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 Processor -- start 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 -- start start 2023-07-14T18:23:03.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 0x7fe3541958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354100e00 0x7fe354195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35419d450 con 0x7fe354101790 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35419d590 con 0x7fe354100e00 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35c6bb700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35419d720 con 0x7fe3540ffc10 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:03.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46810/0 (socket says 172.21.15.64:46810) 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 -- 172.21.15.64:0/3047217816 learned_addr learned my addr 172.21.15.64:0/3047217816 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35a457700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 0x7fe3541958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe359c56700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354100e00 0x7fe354195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 -- 172.21.15.64:0/3047217816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 msgr2=0x7fe3541958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 0x7fe3541958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 -- 172.21.15.64:0/3047217816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354100e00 msgr2=0x7fe354195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:03.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354100e00 0x7fe354195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 -- 172.21.15.64:0/3047217816 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3480095f0 con 0x7fe354101790 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe35ac58700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00e940 tx=0x7fe34c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.561+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c0097b0 con 0x7fe354101790 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.562+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe34c005020 con 0x7fe354101790 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.562+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35419da00 con 0x7fe354101790 2023-07-14T18:23:03.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.562+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe34c00cb70 con 0x7fe354101790 2023-07-14T18:23:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.562+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35419dec0 con 0x7fe354101790 2023-07-14T18:23:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.562+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe354106bf0 con 0x7fe354101790 2023-07-14T18:23:03.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.564+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe34c00cd90 con 0x7fe354101790 2023-07-14T18:23:03.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.564+0000 7fe3537fe700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 0x7fe33c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:03.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.564+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7fe34c014070 con 0x7fe354101790 2023-07-14T18:23:03.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.564+0000 7fe35a457700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 0x7fe33c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:03.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.565+0000 7fe35a457700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 0x7fe33c0a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe344009550 tx=0x7fe3440091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:03.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.568+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe34c096400 con 0x7fe354101790 2023-07-14T18:23:03.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:03.777+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fe35402c8e0 con 0x7fe354101790 2023-07-14T18:23:04.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:23:04.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:04.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:04.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:04.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:04.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:23:04.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:23:04.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:04 smithi105 ceph-mon[98013]: pgmap v247: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:04.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.402+0000 7fe3537fe700 1 -- 172.21.15.64:0/3047217816 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v79) v1 ==== 171+0+0 (secure 0 0 0) 0x7fe34c005190 con 0x7fe354101790 2023-07-14T18:23:04.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.406+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 msgr2=0x7fe33c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:04.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.406+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 0x7fe33c0a3600 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe344009550 tx=0x7fe3440091b0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.406+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 msgr2=0x7fe35419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:04.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.406+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe34c00e940 tx=0x7fe34c00ec50 comp rx=0 tx=0).stop 2023-07-14T18:23:04.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 shutdown_connections 2023-07-14T18:23:04.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe3540ffc10 0x7fe3541958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe33c0a1150 0x7fe33c0a3600 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe344009550 tx=0x7fe3440091b0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe354101790 0x7fe35419ce80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 --2- 172.21.15.64:0/3047217816 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe354100e00 0x7fe354195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.407+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 >> 172.21.15.64:0/3047217816 conn(0x7fe3540fb210 msgr2=0x7fe3540fd540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:04.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.413+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 shutdown_connections 2023-07-14T18:23:04.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.417+0000 7fe35c6bb700 1 -- 172.21.15.64:0/3047217816 wait complete. 2023-07-14T18:23:04.476 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' created 2023-07-14T18:23:04.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T18:23:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:23:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:04.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:04.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:04.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:04.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:23:04.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:23:04.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:04 smithi064 ceph-mon[96720]: pgmap v247: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:04.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 -- 172.21.15.64:0/3934144384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d4100de0 msgr2=0x7fe2d4103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:04.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 --2- 172.21.15.64:0/3934144384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d4100de0 0x7fe2d4103260 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe2c8009990 tx=0x7fe2c8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 -- 172.21.15.64:0/3934144384 shutdown_connections 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 --2- 172.21.15.64:0/3934144384 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d40fe480 0x7fe2d41008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 --2- 172.21.15.64:0/3934144384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d41037a0 0x7fe2d4103c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 --2- 172.21.15.64:0/3934144384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d4100de0 0x7fe2d4103260 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 -- 172.21.15.64:0/3934144384 >> 172.21.15.64:0/3934144384 conn(0x7fe2d40fa130 msgr2=0x7fe2d40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 -- 172.21.15.64:0/3934144384 shutdown_connections 2023-07-14T18:23:04.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.583+0000 7fe2da305700 1 -- 172.21.15.64:0/3934144384 wait complete. 2023-07-14T18:23:04.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 Processor -- start 2023-07-14T18:23:04.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 -- start start 2023-07-14T18:23:04.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4100de0 0x7fe2d419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d41037a0 0x7fe2d41a3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a3af0 con 0x7fe2d40fe480 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a3c30 con 0x7fe2d4100de0 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2da305700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d41a3dc0 con 0x7fe2d41037a0 2023-07-14T18:23:04.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.584+0000 7fe2d3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:04.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46824/0 (socket says 172.21.15.64:46824) 2023-07-14T18:23:04.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 -- 172.21.15.64:0/1398169454 learned_addr learned my addr 172.21.15.64:0/1398169454 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:04.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d88a2700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d41037a0 0x7fe2d41a3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:04.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d37fe700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4100de0 0x7fe2d419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:04.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 -- 172.21.15.64:0/1398169454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d41037a0 msgr2=0x7fe2d41a3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:04.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d41037a0 0x7fe2d41a3520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 -- 172.21.15.64:0/1398169454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4100de0 msgr2=0x7fe2d419c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:04.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4100de0 0x7fe2d419c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:04.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 -- 172.21.15.64:0/1398169454 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c80095f0 con 0x7fe2d40fe480 2023-07-14T18:23:04.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d3fff700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400d950 tx=0x7fe2c400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:04.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400f890 con 0x7fe2d40fe480 2023-07-14T18:23:04.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d41a40a0 con 0x7fe2d40fe480 2023-07-14T18:23:04.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe2c400fe80 con 0x7fe2d40fe480 2023-07-14T18:23:04.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2c400e3b0 con 0x7fe2d40fe480 2023-07-14T18:23:04.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.585+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d41a4680 con 0x7fe2d40fe480 2023-07-14T18:23:04.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.586+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d40fbca0 con 0x7fe2d40fe480 2023-07-14T18:23:04.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.588+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe2c4010460 con 0x7fe2d40fe480 2023-07-14T18:23:04.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.588+0000 7fe2d17fa700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:04.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.588+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fe2c40cc4d0 con 0x7fe2d40fe480 2023-07-14T18:23:04.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.588+0000 7fe2d37fe700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:04.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.589+0000 7fe2d37fe700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800c010 tx=0x7fe2c8005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:04.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.592+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe2c4095dd0 con 0x7fe2d40fe480 2023-07-14T18:23:04.681 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T18:23:04.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:04.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:04.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:04.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:23:04.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1437982698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T18:23:04.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T18:23:04.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:04 smithi144 ceph-mon[98417]: pgmap v247: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:04.914 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:04.863+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe2d402c8e0 con 0x7fe2d40fe480 2023-07-14T18:23:05.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.416+0000 7fe2d17fa700 1 -- 172.21.15.64:0/1398169454 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe2c40160c0 con 0x7fe2d40fe480 2023-07-14T18:23:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.418+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 msgr2=0x7fe2bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.418+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe2c800c010 tx=0x7fe2c8005c60 comp rx=0 tx=0).stop 2023-07-14T18:23:05.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.418+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 msgr2=0x7fe2d419bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.418+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe2c400d950 tx=0x7fe2c400dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 shutdown_connections 2023-07-14T18:23:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe2d41037a0 0x7fe2d41a3520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe2bc0a1150 0x7fe2bc0a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe2d40fe480 0x7fe2d419bf70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 --2- 172.21.15.64:0/1398169454 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe2d4100de0 0x7fe2d419c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 >> 172.21.15.64:0/1398169454 conn(0x7fe2d40fa130 msgr2=0x7fe2d40ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 shutdown_connections 2023-07-14T18:23:05.472 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.419+0000 7fe2da305700 1 -- 172.21.15.64:0/1398169454 wait complete. 2023-07-14T18:23:05.473 INFO:teuthology.orchestra.run.smithi064.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T18:23:05.488 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-14T18:23:05.489 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-14T18:23:05.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:05 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:23:05.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:05 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:23:05.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:05 smithi105 ceph-mon[98013]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:23:05.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:05 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:23:05.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:05 smithi105 ceph-mon[98013]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:23:05.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:05 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:23:05.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:05 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:23:05.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:05 smithi064 ceph-mon[96720]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:23:05.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:05 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:23:05.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:05 smithi064 ceph-mon[96720]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:23:05.626 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 -- 172.21.15.64:0/1262458243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80fed40 msgr2=0x7f85f80ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:05.626 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 --2- 172.21.15.64:0/1262458243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80fed40 0x7f85f80ff1b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009990 tx=0x7f85e8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.626 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 -- 172.21.15.64:0/1262458243 shutdown_connections 2023-07-14T18:23:05.626 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 --2- 172.21.15.64:0/1262458243 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f8100180 0x7f85f8100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.627 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 --2- 172.21.15.64:0/1262458243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f80ff6f0 0x7f85f8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.627 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 --2- 172.21.15.64:0/1262458243 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80fed40 0x7f85f80ff1b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.627 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 -- 172.21.15.64:0/1262458243 >> 172.21.15.64:0/1262458243 conn(0x7f85f80f98a0 msgr2=0x7f85f80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:05.627 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 -- 172.21.15.64:0/1262458243 shutdown_connections 2023-07-14T18:23:05.627 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.574+0000 7f85fd066700 1 -- 172.21.15.64:0/1262458243 wait complete. 2023-07-14T18:23:05.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 Processor -- start 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 -- start start 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f80fed40 0x7f85f81947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f8100180 0x7f85f819bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f819c320 con 0x7f85f80ff6f0 2023-07-14T18:23:05.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f819c460 con 0x7f85f8100180 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85fd066700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f819c5f0 con 0x7f85f80fed40 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85f659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85f659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46844/0 (socket says 172.21.15.64:46844) 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85f659c700 1 -- 172.21.15.64:0/3515261874 learned_addr learned my addr 172.21.15.64:0/3515261874 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.575+0000 7f85f6d9d700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f80fed40 0x7f85f81947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f759e700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f8100180 0x7f85f819bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:05.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 -- 172.21.15.64:0/3515261874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f80fed40 msgr2=0x7f85f81947a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f80fed40 0x7f85f81947a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 -- 172.21.15.64:0/3515261874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f8100180 msgr2=0x7f85f819bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f8100180 0x7f85f819bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 -- 172.21.15.64:0/3515261874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e80095f0 con 0x7f85f80ff6f0 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85f659c700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f85e80059d0 tx=0x7f85e8004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e801d070 con 0x7f85f80ff6f0 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85e800f460 con 0x7f85f80ff6f0 2023-07-14T18:23:05.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f80fdc30 con 0x7f85f80ff6f0 2023-07-14T18:23:05.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8021670 con 0x7f85f80ff6f0 2023-07-14T18:23:05.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.576+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f80fe0e0 con 0x7f85f80ff6f0 2023-07-14T18:23:05.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.577+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f80fead0 con 0x7f85f80ff6f0 2023-07-14T18:23:05.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.578+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f85e800f5d0 con 0x7f85f80ff6f0 2023-07-14T18:23:05.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.579+0000 7f85dffff700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 0x7f85e00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:05.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.579+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f85e80cdcf0 con 0x7f85f80ff6f0 2023-07-14T18:23:05.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.579+0000 7f85f6d9d700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 0x7f85e00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:05.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.579+0000 7f85f6d9d700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 0x7f85e00a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009c60 tx=0x7f85ec010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:05.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.584+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f85e8097580 con 0x7f85f80ff6f0 2023-07-14T18:23:05.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:05 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T18:23:05.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:05 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3047217816' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T18:23:05.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:05 smithi144 ceph-mon[98417]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T18:23:05.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:05 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T18:23:05.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:05 smithi144 ceph-mon[98417]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T18:23:05.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:05.822+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f85f8105520 con 0x7f85f80ff6f0 2023-07-14T18:23:06.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.235+0000 7f85dffff700 1 -- 172.21.15.64:0/3515261874 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) v1 ==== 157+0+0 (secure 0 0 0) 0x7f85e809aba0 con 0x7f85f80ff6f0 2023-07-14T18:23:06.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 msgr2=0x7f85e00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:06.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 0x7f85e00a3680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f85ec009c60 tx=0x7f85ec010040 comp rx=0 tx=0).stop 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 msgr2=0x7f85f8194ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f85e80059d0 tx=0x7f85e8004790 comp rx=0 tx=0).stop 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 shutdown_connections 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f85f80fed40 0x7f85f81947a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f85e00a11d0 0x7f85e00a3680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.237+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f85f8100180 0x7f85f819bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.238+0000 7f85fd066700 1 --2- 172.21.15.64:0/3515261874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85f80ff6f0 0x7f85f8194ce0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.238+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 >> 172.21.15.64:0/3515261874 conn(0x7f85f80f98a0 msgr2=0x7f85f80fbbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:06.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.238+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 shutdown_connections 2023-07-14T18:23:06.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.238+0000 7f85fd066700 1 -- 172.21.15.64:0/3515261874 wait complete. 2023-07-14T18:23:06.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T18:23:06.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.386+0000 7f0aa78a0700 1 -- 172.21.15.64:0/2613875162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00ffd40 msgr2=0x7f0aa0100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:06.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.386+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/2613875162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0100190 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a90009990 tx=0x7f0a90009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- 172.21.15.64:0/2613875162 shutdown_connections 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/2613875162 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa01006d0 0x7f0aa0106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/2613875162 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa00feb50 0x7f0aa00fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/2613875162 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0100190 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- 172.21.15.64:0/2613875162 >> 172.21.15.64:0/2613875162 conn(0x7f0aa00fa150 msgr2=0x7f0aa00fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- 172.21.15.64:0/2613875162 shutdown_connections 2023-07-14T18:23:06.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- 172.21.15.64:0/2613875162 wait complete. 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 Processor -- start 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- start start 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa01006d0 0x7f0aa019cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:06.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa019d350 con 0x7f0aa00feb50 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa019d490 con 0x7f0aa01006d0 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.387+0000 7f0aa78a0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa019d620 con 0x7f0aa00ffd40 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46856/0 (socket says 172.21.15.64:46856) 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 -- 172.21.15.64:0/3008033112 learned_addr learned my addr 172.21.15.64:0/3008033112 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa5e3d700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa01006d0 0x7f0aa019cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:06.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa4e3b700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 -- 172.21.15.64:0/3008033112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 msgr2=0x7f0aa0195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 -- 172.21.15.64:0/3008033112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa01006d0 msgr2=0x7f0aa019cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa01006d0 0x7f0aa019cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 -- 172.21.15.64:0/3008033112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900095f0 con 0x7f0aa00feb50 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa4e3b700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:06.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa563c700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00d980 tx=0x7f0a9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:06.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c004020 con 0x7f0aa00feb50 2023-07-14T18:23:06.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa019d900 con 0x7f0aa00feb50 2023-07-14T18:23:06.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0a9c009730 con 0x7f0aa00feb50 2023-07-14T18:23:06.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9c0039c0 con 0x7f0aa00feb50 2023-07-14T18:23:06.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.388+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa019de60 con 0x7f0aa00feb50 2023-07-14T18:23:06.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.389+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa0106b30 con 0x7f0aa00feb50 2023-07-14T18:23:06.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.390+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0a9c0098a0 con 0x7f0aa00feb50 2023-07-14T18:23:06.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.391+0000 7f0a967fc700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 0x7f0a8c0a36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:06.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.391+0000 7f0aa4e3b700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 0x7f0a8c0a36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:06.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.391+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f0a9c0cc980 con 0x7f0aa00feb50 2023-07-14T18:23:06.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.391+0000 7f0aa4e3b700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 0x7f0a8c0a36a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a900059d0 tx=0x7f0a90005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:06.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.394+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0a9c0960f0 con 0x7f0aa00feb50 2023-07-14T18:23:06.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:23:06.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:23:06.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:23:06.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:23:06.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: pgmap v250: 97 pgs: 12 active+clean, 10 creating+peering, 75 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:06.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:23:06.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.563 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:06.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:06.564 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:23:06.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:23:06.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:23:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:23:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: pgmap v250: 97 pgs: 12 active+clean, 10 creating+peering, 75 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:23:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.595 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:06.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:06.596 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:06.597+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0aa002cb90 con 0x7f0aa00feb50 2023-07-14T18:23:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T18:23:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1398169454' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T18:23:06.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: pgmap v250: 97 pgs: 12 active+clean, 10 creating+peering, 75 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:06.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:23:06.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:23:06.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:23:07.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:23:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:23:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:active 2023-07-14T18:23:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:07 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:23:07.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:23:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:23:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:active 2023-07-14T18:23:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:07 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:23:07.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3515261874' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T18:23:07.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T18:23:07.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] up:active 2023-07-14T18:23:07.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:07.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:07 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T18:23:08.490 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.438+0000 7f0a967fc700 1 -- 172.21.15.64:0/3008033112 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) v1 ==== 228+0+0 (secure 0 0 0) 0x7f0a9c099710 con 0x7f0aa00feb50 2023-07-14T18:23:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.444+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 msgr2=0x7f0a8c0a36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:08.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.444+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 0x7f0a8c0a36a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0a900059d0 tx=0x7f0a90005960 comp rx=0 tx=0).stop 2023-07-14T18:23:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.444+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 msgr2=0x7f0aa01957d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.444+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c00d980 tx=0x7f0a9c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:23:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 shutdown_connections 2023-07-14T18:23:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0a8c0a11f0 0x7f0a8c0a36a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0aa01006d0 0x7f0aa019cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0aa00feb50 0x7f0aa01957d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 --2- 172.21.15.64:0/3008033112 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0aa00ffd40 0x7f0aa0195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 >> 172.21.15.64:0/3008033112 conn(0x7f0aa00fa150 msgr2=0x7f0aa00fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 shutdown_connections 2023-07-14T18:23:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.445+0000 7f0aa78a0700 1 -- 172.21.15.64:0/3008033112 wait complete. 2023-07-14T18:23:08.500 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data_ec' created 2023-07-14T18:23:08.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-14T18:23:08.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/3042601113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 msgr2=0x7f0d24101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:08.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/3042601113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24101250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d20009990 tx=0x7f0d20009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/3042601113 shutdown_connections 2023-07-14T18:23:08.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/3042601113 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d24101790 0x7f0d24108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/3042601113 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d240ffc10 0x7f0d24100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/3042601113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24101250 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/3042601113 >> 172.21.15.64:0/3042601113 conn(0x7f0d240fb250 msgr2=0x7f0d240fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:08.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/3042601113 shutdown_connections 2023-07-14T18:23:08.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.597+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/3042601113 wait complete. 2023-07-14T18:23:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 Processor -- start 2023-07-14T18:23:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 -- start start 2023-07-14T18:23:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d240ffc10 0x7f0d24195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:08.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d24101790 0x7f0d2419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2419d4a0 con 0x7f0d24100e00 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2419d5e0 con 0x7f0d24101790 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2bbcb700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2419d770 con 0x7f0d240ffc10 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d29166700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d29967700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d240ffc10 0x7f0d24195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d29166700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46872/0 (socket says 172.21.15.64:46872) 2023-07-14T18:23:08.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d29166700 1 -- 172.21.15.64:0/2983139030 learned_addr learned my addr 172.21.15.64:0/2983139030 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.598+0000 7f0d2a168700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d24101790 0x7f0d2419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 -- 172.21.15.64:0/2983139030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d240ffc10 msgr2=0x7f0d24195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d240ffc10 0x7f0d24195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 -- 172.21.15.64:0/2983139030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d24101790 msgr2=0x7f0d2419ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d24101790 0x7f0d2419ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 -- 172.21.15.64:0/2983139030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d200095f0 con 0x7f0d24100e00 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d29166700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d200098e0 tx=0x7f0d20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:08.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2001d070 con 0x7f0d24100e00 2023-07-14T18:23:08.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0d2000bdd0 con 0x7f0d24100e00 2023-07-14T18:23:08.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2419d9f0 con 0x7f0d24100e00 2023-07-14T18:23:08.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20022d50 con 0x7f0d24100e00 2023-07-14T18:23:08.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.599+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2419de90 con 0x7f0d24100e00 2023-07-14T18:23:08.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.600+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d2410b360 con 0x7f0d24100e00 2023-07-14T18:23:08.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.601+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0d2000b940 con 0x7f0d24100e00 2023-07-14T18:23:08.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.602+0000 7f0d16ffd700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:08.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.602+0000 7f0d29967700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:08.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.602+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f0d200ce4e0 con 0x7f0d24100e00 2023-07-14T18:23:08.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.603+0000 7f0d29967700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 0x7f0d100a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d24100c60 tx=0x7f0d18009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:08.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.606+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0d20097af0 con 0x7f0d24100e00 2023-07-14T18:23:08.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:08 smithi105 ceph-mon[98013]: pgmap v253: 97 pgs: 36 active+clean, 10 creating+peering, 51 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2023-07-14T18:23:08.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:08 smithi105 ceph-mon[98013]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:23:08.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:08 smithi064 ceph-mon[96720]: pgmap v253: 97 pgs: 36 active+clean, 10 creating+peering, 51 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2023-07-14T18:23:08.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:08 smithi064 ceph-mon[96720]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:23:08.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:08.803+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f0d24106bc0 con 0x7f0d24100e00 2023-07-14T18:23:08.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:08 smithi144 ceph-mon[98417]: pgmap v253: 97 pgs: 36 active+clean, 10 creating+peering, 51 unknown; 578 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 6 op/s 2023-07-14T18:23:08.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:08 smithi144 ceph-mon[98417]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T18:23:09.489 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.434+0000 7f0d16ffd700 1 -- 172.21.15.64:0/2983139030 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f0d2009b110 con 0x7f0d24100e00 2023-07-14T18:23:09.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.441+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 msgr2=0x7f0d100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:09.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.441+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 0x7f0d100a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0d24100c60 tx=0x7f0d18009260 comp rx=0 tx=0).stop 2023-07-14T18:23:09.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.441+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 msgr2=0x7f0d24195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:09.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.441+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0d200098e0 tx=0x7f0d20004730 comp rx=0 tx=0).stop 2023-07-14T18:23:09.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 shutdown_connections 2023-07-14T18:23:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0d240ffc10 0x7f0d24195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0d100a1150 0x7f0d100a3600 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0d24101790 0x7f0d2419ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 --2- 172.21.15.64:0/2983139030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d24100e00 0x7f0d24195e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.446+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 >> 172.21.15.64:0/2983139030 conn(0x7f0d240fb250 msgr2=0x7f0d240fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:09.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.447+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 shutdown_connections 2023-07-14T18:23:09.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.448+0000 7f0d2bbcb700 1 -- 172.21.15.64:0/2983139030 wait complete. 2023-07-14T18:23:09.502 INFO:teuthology.orchestra.run.smithi064.stderr:set pool 4 allow_ec_overwrites to true 2023-07-14T18:23:09.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-14T18:23:09.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 -- 172.21.15.64:0/2796667207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00ffdc0 msgr2=0x7f6ba0100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/2796667207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0100230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009990 tx=0x7f6b9c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 -- 172.21.15.64:0/2796667207 shutdown_connections 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/2796667207 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba00feb30 0x7f6ba00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/2796667207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba0100770 0x7f6ba0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/2796667207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0100230 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 -- 172.21.15.64:0/2796667207 >> 172.21.15.64:0/2796667207 conn(0x7f6ba00fa150 msgr2=0x7f6ba00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:09.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 -- 172.21.15.64:0/2796667207 shutdown_connections 2023-07-14T18:23:09.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 -- 172.21.15.64:0/2796667207 wait complete. 2023-07-14T18:23:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.595+0000 7f6ba87f3700 1 Processor -- start 2023-07-14T18:23:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 -- start start 2023-07-14T18:23:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:09.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 0x7f6ba019bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba019c2f0 con 0x7f6ba00feb30 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba019c430 con 0x7f6ba00ffdc0 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba87f3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba019c5c0 con 0x7f6ba0100770 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46896/0 (socket says 172.21.15.64:46896) 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba5d8e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 -- 172.21.15.64:0/3663306772 learned_addr learned my addr 172.21.15.64:0/3663306772 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:09.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba6d90700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 0x7f6ba019bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 -- 172.21.15.64:0/3663306772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 msgr2=0x7f6ba019bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 0x7f6ba019bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 -- 172.21.15.64:0/3663306772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 msgr2=0x7f6ba0194cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0194cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 -- 172.21.15.64:0/3663306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c0095f0 con 0x7f6ba00feb30 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba6d90700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 0x7f6ba019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:09.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba5d8e700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6ba658f700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400d950 tx=0x7f6b9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.596+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94009940 con 0x7f6ba00feb30 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.597+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6b94010460 con 0x7f6ba00feb30 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.597+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba019c8a0 con 0x7f6ba00feb30 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.597+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400f620 con 0x7f6ba00feb30 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.597+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba019ce00 con 0x7f6ba00feb30 2023-07-14T18:23:09.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.598+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba010a1b0 con 0x7f6ba00feb30 2023-07-14T18:23:09.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.599+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6b940105d0 con 0x7f6ba00feb30 2023-07-14T18:23:09.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.599+0000 7f6b937fe700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 0x7f6b8c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:09.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.599+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f6b940cc850 con 0x7f6ba00feb30 2023-07-14T18:23:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.600+0000 7f6ba5d8e700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 0x7f6b8c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.600+0000 7f6ba5d8e700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 0x7f6b8c0a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00b5c0 tx=0x7f6b9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:09.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.603+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6b94095dd0 con 0x7f6ba00feb30 2023-07-14T18:23:09.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:09 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:23:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:09 smithi105 ceph-mon[98013]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:23:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:09 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:23:09.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:09 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:23:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:09 smithi064 ceph-mon[96720]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:23:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:09 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:23:09.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:09.851+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f6ba0105b60 con 0x7f6ba00feb30 2023-07-14T18:23:09.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:09 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3008033112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T18:23:09.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:09 smithi144 ceph-mon[98417]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T18:23:09.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:09 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T18:23:10.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.452+0000 7f6b937fe700 1 -- 172.21.15.64:0/3663306772 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f6b940200b0 con 0x7f6ba00feb30 2023-07-14T18:23:10.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.454+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 msgr2=0x7f6b8c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.454+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 0x7f6b8c0a3680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00b5c0 tx=0x7f6b9c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.454+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 msgr2=0x7f6ba0194770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.454+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6b9400d950 tx=0x7f6b9400dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 shutdown_connections 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6ba0100770 0x7f6ba019bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6b8c0a11d0 0x7f6b8c0a3680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6ba00feb30 0x7f6ba0194770 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 --2- 172.21.15.64:0/3663306772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6ba00ffdc0 0x7f6ba0194cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 >> 172.21.15.64:0/3663306772 conn(0x7f6ba00fa150 msgr2=0x7f6ba00fc410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:10.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 shutdown_connections 2023-07-14T18:23:10.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.455+0000 7f6ba87f3700 1 -- 172.21.15.64:0/3663306772 wait complete. 2023-07-14T18:23:10.508 INFO:teuthology.orchestra.run.smithi064.stderr:added data pool 4 to fsmap 2023-07-14T18:23:10.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:23:10.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 -- 172.21.15.64:0/1410348748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f00ffcf0 msgr2=0x7f55f0100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/1410348748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f00ffcf0 0x7f55f0100140 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f55e0009990 tx=0x7f55e0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 -- 172.21.15.64:0/1410348748 shutdown_connections 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/1410348748 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f0100680 0x7f55f0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/1410348748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/1410348748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f00ffcf0 0x7f55f0100140 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.602+0000 7f55f7b4d700 1 -- 172.21.15.64:0/1410348748 >> 172.21.15.64:0/1410348748 conn(0x7f55f00fa100 msgr2=0x7f55f00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- 172.21.15.64:0/1410348748 shutdown_connections 2023-07-14T18:23:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- 172.21.15.64:0/1410348748 wait complete. 2023-07-14T18:23:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 Processor -- start 2023-07-14T18:23:10.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- start start 2023-07-14T18:23:10.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:10.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 0x7f55f019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:10.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:10.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f019d300 con 0x7f55f0100680 2023-07-14T18:23:10.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f019d470 con 0x7f55f00feb00 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f7b4d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f019d630 con 0x7f55f00ffcf0 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.603+0000 7f55f60ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46912/0 (socket says 172.21.15.64:46912) 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 -- 172.21.15.64:0/3497587466 learned_addr learned my addr 172.21.15.64:0/3497587466 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f50e8700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 0x7f55f019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f58e9700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 -- 172.21.15.64:0/3497587466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 msgr2=0x7f55f019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 0x7f55f019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 -- 172.21.15.64:0/3497587466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 msgr2=0x7f55f0068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f0068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 -- 172.21.15.64:0/3497587466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55e00095f0 con 0x7f55f0100680 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f58e9700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f50e8700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 0x7f55f019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f60ea700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f55e8007d70 tx=0x7f55e800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:10.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e800ad40 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55e80128b0 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80116b0 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f019d910 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.604+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f019ddd0 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.605+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f010a1d0 con 0x7f55f0100680 2023-07-14T18:23:10.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.606+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f55e8012a20 con 0x7f55f0100680 2023-07-14T18:23:10.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.607+0000 7f55e6ffd700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 0x7f55dc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:10.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.607+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f55e80cd190 con 0x7f55f0100680 2023-07-14T18:23:10.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.607+0000 7f55f58e9700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 0x7f55dc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:10.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.607+0000 7f55f58e9700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 0x7f55dc0a36d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55ec005fd0 tx=0x7f55ec005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:10.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.610+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55e80966f0 con 0x7f55f0100680 2023-07-14T18:23:10.811 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:23:10 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:23:10.277+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T18:23:10.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:10 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:23:10.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:10 smithi105 ceph-mon[98013]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:23:10.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:10 smithi105 ceph-mon[98013]: pgmap v256: 129 pgs: 95 active+clean, 9 creating+peering, 25 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-14T18:23:10.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:10 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:23:10.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:10 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:23:10.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:10 smithi064 ceph-mon[96720]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:23:10.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:10 smithi064 ceph-mon[96720]: pgmap v256: 129 pgs: 95 active+clean, 9 creating+peering, 25 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-14T18:23:10.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:10 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:23:10.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.855+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55f010a3c0 con 0x7f55f0100680 2023-07-14T18:23:10.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.855+0000 7f55e6ffd700 1 -- 172.21.15.64:0/3497587466 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10781 (secure 0 0 0) 0x7f55e8016020 con 0x7f55f0100680 2023-07-14T18:23:10.907 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:10.907 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:10.440958+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T18:23:10.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 msgr2=0x7f55dc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 0x7f55dc0a36d0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f55ec005fd0 tx=0x7f55ec005f00 comp rx=0 tx=0).stop 2023-07-14T18:23:10.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 msgr2=0x7f55f019cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f55e8007d70 tx=0x7f55e800a340 comp rx=0 tx=0).stop 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 shutdown_connections 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f55dc0a1220 0x7f55dc0a36d0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f55f0100680 0x7f55f019cb60 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f55f00feb00 0x7f55f0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 --2- 172.21.15.64:0/3497587466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f55f00ffcf0 0x7f55f019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:10.910 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 >> 172.21.15.64:0/3497587466 conn(0x7f55f00fa100 msgr2=0x7f55f00fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:10.911 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 shutdown_connections 2023-07-14T18:23:10.911 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.857+0000 7f55f7b4d700 1 -- 172.21.15.64:0/3497587466 wait complete. 2023-07-14T18:23:10.911 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 17 2023-07-14T18:23:10.921 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:10.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:10 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2983139030' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T18:23:10.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:10 smithi144 ceph-mon[98417]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T18:23:10.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:10 smithi144 ceph-mon[98417]: pgmap v256: 129 pgs: 95 active+clean, 9 creating+peering, 25 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2023-07-14T18:23:10.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:10 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T18:23:11.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3461128264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 msgr2=0x7fea38106fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3461128264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38106fa0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fea3000b0a0 tx=0x7fea3000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3461128264 shutdown_connections 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3461128264 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38106fa0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3461128264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea38100180 0x7fea38100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3461128264 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea380fed40 0x7fea380ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3461128264 >> 172.21.15.64:0/3461128264 conn(0x7fea380f98a0 msgr2=0x7fea380fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3461128264 shutdown_connections 2023-07-14T18:23:11.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3461128264 wait complete. 2023-07-14T18:23:11.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.996+0000 7fea3d7ce700 1 Processor -- start 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 -- start start 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 0x7fea38102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38100180 0x7fea38101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38101e10 con 0x7fea380ff6f0 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38101f80 con 0x7fea380fed40 2023-07-14T18:23:11.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3819f9b0 con 0x7fea38100180 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46926/0 (socket says 172.21.15.64:46926) 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 -- 172.21.15.64:0/3316625483 learned_addr learned my addr 172.21.15.64:0/3316625483 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea377fe700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38100180 0x7fea38101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea36ffd700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 0x7fea38102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 -- 172.21.15.64:0/3316625483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38100180 msgr2=0x7fea38101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38100180 0x7fea38101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 -- 172.21.15.64:0/3316625483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 msgr2=0x7fea38102af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 0x7fea38102af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 -- 172.21.15.64:0/3316625483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30009d00 con 0x7fea380ff6f0 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea36ffd700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 0x7fea38102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea367fc700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea2800d950 tx=0x7fea2800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28009940 con 0x7fea380ff6f0 2023-07-14T18:23:11.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.997+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3819fc90 con 0x7fea380ff6f0 2023-07-14T18:23:11.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.998+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fea28010460 con 0x7fea380ff6f0 2023-07-14T18:23:11.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.998+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2800f620 con 0x7fea380ff6f0 2023-07-14T18:23:11.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.998+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea381a01c0 con 0x7fea380ff6f0 2023-07-14T18:23:11.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:10.999+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3810a1d0 con 0x7fea380ff6f0 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.000+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fea280105d0 con 0x7fea380ff6f0 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.000+0000 7fea1ffff700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 0x7fea200a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.000+0000 7fea36ffd700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 0x7fea200a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.000+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fea280cc970 con 0x7fea380ff6f0 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.001+0000 7fea36ffd700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 0x7fea200a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea2c005d90 tx=0x7fea2c005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.004+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fea28095ed0 con 0x7fea380ff6f0 2023-07-14T18:23:11.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.203+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea381a0770 con 0x7fea380ff6f0 2023-07-14T18:23:11.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.205+0000 7fea1ffff700 1 -- 172.21.15.64:0/3316625483 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7fea28016090 con 0x7fea380ff6f0 2023-07-14T18:23:11.257 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:11.258 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:11.259 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 msgr2=0x7fea200a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.260 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 0x7fea200a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea2c005d90 tx=0x7fea2c005d00 comp rx=0 tx=0).stop 2023-07-14T18:23:11.260 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 msgr2=0x7fea38101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.260 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fea2800d950 tx=0x7fea2800dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:11.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 shutdown_connections 2023-07-14T18:23:11.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fea38100180 0x7fea38101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fea200a11a0 0x7fea200a3650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea380fed40 0x7fea38102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.261 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 --2- 172.21.15.64:0/3316625483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fea380ff6f0 0x7fea38101130 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.262 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 >> 172.21.15.64:0/3316625483 conn(0x7fea380f98a0 msgr2=0x7fea380fa0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:11.262 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 shutdown_connections 2023-07-14T18:23:11.262 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.207+0000 7fea3d7ce700 1 -- 172.21.15.64:0/3316625483 wait complete. 2023-07-14T18:23:11.273 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:11.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 -- 172.21.15.64:0/3184911683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980fee50 msgr2=0x7f3e980ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/3184911683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980fee50 0x7f3e980ff2c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009990 tx=0x7f3e8c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.404 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 -- 172.21.15.64:0/3184911683 shutdown_connections 2023-07-14T18:23:11.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/3184911683 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980ff800 0x7f3e980ffc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/3184911683 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 0x7f3e980fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/3184911683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980fee50 0x7f3e980ff2c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 -- 172.21.15.64:0/3184911683 >> 172.21.15.64:0/3184911683 conn(0x7f3e980fa130 msgr2=0x7f3e980fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:11.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 -- 172.21.15.64:0/3184911683 shutdown_connections 2023-07-14T18:23:11.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.349+0000 7f3e9da2d700 1 -- 172.21.15.64:0/3184911683 wait complete. 2023-07-14T18:23:11.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 Processor -- start 2023-07-14T18:23:11.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 -- start start 2023-07-14T18:23:11.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 0x7f3e981979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 0x7f3e98197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9819f560 con 0x7f3e980ff800 2023-07-14T18:23:11.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9819f6a0 con 0x7f3e980fe500 2023-07-14T18:23:11.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e9da2d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9819f830 con 0x7f3e980fee50 2023-07-14T18:23:11.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e977fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e977fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46942/0 (socket says 172.21.15.64:46942) 2023-07-14T18:23:11.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e977fe700 1 -- 172.21.15.64:0/1851793612 learned_addr learned my addr 172.21.15.64:0/1851793612 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:11.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.350+0000 7f3e96ffd700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 0x7f3e981979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e967fc700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 0x7f3e98197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 -- 172.21.15.64:0/1851793612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 msgr2=0x7f3e98197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 0x7f3e98197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 -- 172.21.15.64:0/1851793612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 msgr2=0x7f3e981979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 0x7f3e981979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 -- 172.21.15.64:0/1851793612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0095f0 con 0x7f3e980ff800 2023-07-14T18:23:11.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e967fc700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 0x7f3e98197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:11.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e977fe700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e90007d70 tx=0x7f3e9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9000ad40 con 0x7f3e980ff800 2023-07-14T18:23:11.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e900128b0 con 0x7f3e980ff800 2023-07-14T18:23:11.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9819fb10 con 0x7f3e980ff800 2023-07-14T18:23:11.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e900115a0 con 0x7f3e980ff800 2023-07-14T18:23:11.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.351+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e981a0070 con 0x7f3e980ff800 2023-07-14T18:23:11.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.352+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e981092e0 con 0x7f3e980ff800 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.353+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3e90012a20 con 0x7f3e980ff800 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.354+0000 7f3e9ca2b700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 0x7f3e800a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.354+0000 7f3e96ffd700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 0x7f3e800a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.354+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3e900cd070 con 0x7f3e980ff800 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.354+0000 7f3e96ffd700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 0x7f3e800a36d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e88005950 tx=0x7f3e8800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.357+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e900965d0 con 0x7f3e980ff800 2023-07-14T18:23:11.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:11 smithi064 ceph-mon[96720]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:23:11.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:11 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:23:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:11 smithi064 ceph-mon[96720]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:11 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3497587466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:11 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3316625483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:11.610 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.559+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e9802cb30 con 0x7f3e980ff800 2023-07-14T18:23:11.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.560+0000 7f3e9ca2b700 1 -- 172.21.15.64:0/1851793612 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f3e90016070 con 0x7f3e980ff800 2023-07-14T18:23:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:11.612 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:11.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 msgr2=0x7f3e800a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 0x7f3e800a36d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e88005950 tx=0x7f3e8800b410 comp rx=0 tx=0).stop 2023-07-14T18:23:11.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 msgr2=0x7f3e9819ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f3e90007d70 tx=0x7f3e9000a340 comp rx=0 tx=0).stop 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 shutdown_connections 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f3e800a1220 0x7f3e800a36d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e980ff800 0x7f3e9819ef90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3e980fe500 0x7f3e981979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 --2- 172.21.15.64:0/1851793612 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3e980fee50 0x7f3e98197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 >> 172.21.15.64:0/1851793612 conn(0x7f3e980fa130 msgr2=0x7f3e980fbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:11.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 shutdown_connections 2023-07-14T18:23:11.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.562+0000 7f3e9da2d700 1 -- 172.21.15.64:0/1851793612 wait complete. 2023-07-14T18:23:11.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:23:11.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 -- 172.21.15.64:0/739793189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f01006e0 msgr2=0x7fd2f0102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/739793189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f01006e0 0x7fd2f0102ac0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000b0a0 tx=0x7fd2e000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 -- 172.21.15.64:0/739793189 shutdown_connections 2023-07-14T18:23:11.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/739793189 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f0105920 0x7fd2f0107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/739793189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f01006e0 0x7fd2f0102ac0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/739793189 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0103000 0x7fd2f01053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 -- 172.21.15.64:0/739793189 >> 172.21.15.64:0/739793189 conn(0x7fd2f00fa100 msgr2=0x7fd2f00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:11.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 -- 172.21.15.64:0/739793189 shutdown_connections 2023-07-14T18:23:11.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.726+0000 7fd2f87f9700 1 -- 172.21.15.64:0/739793189 wait complete. 2023-07-14T18:23:11.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 Processor -- start 2023-07-14T18:23:11.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 -- start start 2023-07-14T18:23:11.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f01006e0 0x7fd2f01959c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0105920 0x7fd2f019cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f019d540 con 0x7fd2f0103000 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f019d680 con 0x7fd2f01006e0 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f87f9700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f019d810 con 0x7fd2f0105920 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46954/0 (socket says 172.21.15.64:46954) 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 -- 172.21.15.64:0/3051199466 learned_addr learned my addr 172.21.15.64:0/3051199466 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:11.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f6595700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f01006e0 0x7fd2f01959c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 -- 172.21.15.64:0/3051199466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0105920 msgr2=0x7fd2f019cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0105920 0x7fd2f019cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 -- 172.21.15.64:0/3051199466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f01006e0 msgr2=0x7fd2f01959c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f01006e0 0x7fd2f01959c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 -- 172.21.15.64:0/3051199466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0009d00 con 0x7fd2f0103000 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.727+0000 7fd2f5d94700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c920 tx=0x7fd2e400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.728+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4007b00 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.728+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd2e400ce70 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.728+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f019daf0 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.728+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4005440 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.728+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f019e050 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.729+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f00fbc70 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.730+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd2e40055a0 con 0x7fd2f0103000 2023-07-14T18:23:11.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.730+0000 7fd2ef7fe700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 0x7fd2d80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:11.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.730+0000 7fd2f6595700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 0x7fd2d80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:11.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.731+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd2e40cc650 con 0x7fd2f0103000 2023-07-14T18:23:11.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.731+0000 7fd2f6595700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 0x7fd2d80a36d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009fd0 tx=0x7fd2e000b7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:11.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.735+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd2e4095bb0 con 0x7fd2f0103000 2023-07-14T18:23:11.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:11 smithi105 ceph-mon[98013]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:23:11.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:11 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:23:11.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:11 smithi105 ceph-mon[98013]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:11.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:11 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3497587466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:11.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:11 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3316625483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:11 smithi144 ceph-mon[98417]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T18:23:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:11 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3663306772' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T18:23:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:11 smithi144 ceph-mon[98417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:11 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3497587466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:11 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3316625483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.981+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f002c8e0 con 0x7fd2f0103000 2023-07-14T18:23:12.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.982+0000 7fd2ef7fe700 1 -- 172.21.15.64:0/3051199466 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10781 (secure 0 0 0) 0x7fd2e40991d0 con 0x7fd2f0103000 2023-07-14T18:23:12.034 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:12.034 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14556,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24487,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24500,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24508,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:10.440958+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24545},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T18:23:12.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 msgr2=0x7fd2d80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 0x7fd2d80a36d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0009fd0 tx=0x7fd2e000b7f0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 msgr2=0x7fd2f0195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400c920 tx=0x7fd2e400cc30 comp rx=0 tx=0).stop 2023-07-14T18:23:12.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 shutdown_connections 2023-07-14T18:23:12.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd2f0105920 0x7fd2f019cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd2d80a1220 0x7fd2d80a36d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2f01006e0 0x7fd2f01959c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 --2- 172.21.15.64:0/3051199466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2f0103000 0x7fd2f0195f00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 >> 172.21.15.64:0/3051199466 conn(0x7fd2f00fa100 msgr2=0x7fd2f0101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.984+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 shutdown_connections 2023-07-14T18:23:12.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:11.985+0000 7fd2f87f9700 1 -- 172.21.15.64:0/3051199466 wait complete. 2023-07-14T18:23:12.039 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 17 2023-07-14T18:23:12.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:12.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/2677512744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e4100680 msgr2=0x7fb7e4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/2677512744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e4100680 0x7fb7e4106f40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00b0a0 tx=0x7fb7dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/2677512744 shutdown_connections 2023-07-14T18:23:12.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/2677512744 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e40feb00 0x7fb7e40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/2677512744 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e4100680 0x7fb7e4106f40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/2677512744 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 0x7fb7e4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/2677512744 >> 172.21.15.64:0/2677512744 conn(0x7fb7e40fa100 msgr2=0x7fb7e40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/2677512744 shutdown_connections 2023-07-14T18:23:12.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.123+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/2677512744 wait complete. 2023-07-14T18:23:12.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 Processor -- start 2023-07-14T18:23:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 -- start start 2023-07-14T18:23:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 0x7fb7e419a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 0x7fb7e419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419d460 con 0x7fb7e40feb00 2023-07-14T18:23:12.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419d5d0 con 0x7fb7e40ffcf0 2023-07-14T18:23:12.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e9ac2700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e419d790 con 0x7fb7e4100680 2023-07-14T18:23:12.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46966/0 (socket says 172.21.15.64:46966) 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 -- 172.21.15.64:0/4072309881 learned_addr learned my addr 172.21.15.64:0/4072309881 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e2ffd700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 0x7fb7e419a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e3fff700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 0x7fb7e419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 -- 172.21.15.64:0/4072309881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 msgr2=0x7fb7e419ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 0x7fb7e419ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 -- 172.21.15.64:0/4072309881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 msgr2=0x7fb7e419a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 0x7fb7e419a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e37fe700 1 -- 172.21.15.64:0/4072309881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc009d00 con 0x7fb7e40feb00 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.124+0000 7fb7e3fff700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 0x7fb7e419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e37fe700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00d980 tx=0x7fb7cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc009930 con 0x7fb7e40feb00 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb7cc010460 con 0x7fb7e40feb00 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e419da70 con 0x7fb7e40feb00 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc00f620 con 0x7fb7e40feb00 2023-07-14T18:23:12.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.125+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e419df80 con 0x7fb7e40feb00 2023-07-14T18:23:12.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.126+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e410a1b0 con 0x7fb7e40feb00 2023-07-14T18:23:12.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.127+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb7cc009a90 con 0x7fb7e40feb00 2023-07-14T18:23:12.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.127+0000 7fb7e0ff9700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 0x7fb7d00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.128+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb7cc0cc570 con 0x7fb7e40feb00 2023-07-14T18:23:12.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.128+0000 7fb7e2ffd700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 0x7fb7d00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.128+0000 7fb7e2ffd700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 0x7fb7d00a36d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4007be0 tx=0x7fb7d40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.132+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb7cc095ad0 con 0x7fb7e40feb00 2023-07-14T18:23:12.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.333+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7e410a3d0 con 0x7fb7e40feb00 2023-07-14T18:23:12.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.335+0000 7fb7e0ff9700 1 -- 172.21.15.64:0/4072309881 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7fb7cc0990f0 con 0x7fb7e40feb00 2023-07-14T18:23:12.387 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:12.387 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:12.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 msgr2=0x7fb7d00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 0x7fb7d00a36d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4007be0 tx=0x7fb7d40073d0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 msgr2=0x7fb7e4068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00d980 tx=0x7fb7cc00dc90 comp rx=0 tx=0).stop 2023-07-14T18:23:12.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 shutdown_connections 2023-07-14T18:23:12.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7e4100680 0x7fb7e419ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7d00a1220 0x7fb7d00a36d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7e40feb00 0x7fb7e4068310 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 --2- 172.21.15.64:0/4072309881 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7e40ffcf0 0x7fb7e419a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 >> 172.21.15.64:0/4072309881 conn(0x7fb7e40fa100 msgr2=0x7fb7e40fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 shutdown_connections 2023-07-14T18:23:12.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.337+0000 7fb7e9ac2700 1 -- 172.21.15.64:0/4072309881 wait complete. 2023-07-14T18:23:12.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-14T18:23:04.865157+0000', 'modified': '2023-07-14T18:23:10.440958+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24545}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/2402376208', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 2402376208}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 2402376208}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-14T18:23:12.402 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-14T18:23:12.403 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:23:12.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- 172.21.15.64:0/323090278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 msgr2=0x7ff624106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 --2- 172.21.15.64:0/323090278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff624106f50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7ff61c00b0a0 tx=0x7ff61c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- 172.21.15.64:0/323090278 shutdown_connections 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 --2- 172.21.15.64:0/323090278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff624106f50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 --2- 172.21.15.64:0/323090278 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240feb10 0x7ff6240fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 --2- 172.21.15.64:0/323090278 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240ffd00 0x7ff624100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- 172.21.15.64:0/323090278 >> 172.21.15.64:0/323090278 conn(0x7ff6240fa130 msgr2=0x7ff6240fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- 172.21.15.64:0/323090278 shutdown_connections 2023-07-14T18:23:12.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- 172.21.15.64:0/323090278 wait complete. 2023-07-14T18:23:12.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 Processor -- start 2023-07-14T18:23:12.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.479+0000 7ff629d4f700 1 -- start start 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 0x7ff624195a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240ffd00 0x7ff624195f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62419d5d0 con 0x7ff624100690 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62419d710 con 0x7ff6240ffd00 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62419d8a0 con 0x7ff6240feb10 2023-07-14T18:23:12.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46984/0 (socket says 172.21.15.64:46984) 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 -- 172.21.15.64:0/1730495917 learned_addr learned my addr 172.21.15.64:0/1730495917 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff622ffd700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240ffd00 0x7ff624195f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 0x7ff624195a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 -- 172.21.15.64:0/1730495917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 msgr2=0x7ff624195a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 0x7ff624195a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 -- 172.21.15.64:0/1730495917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240ffd00 msgr2=0x7ff624195f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240ffd00 0x7ff624195f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 -- 172.21.15.64:0/1730495917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c009d00 con 0x7ff624100690 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 0x7ff624195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff623fff700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009fd0 tx=0x7ff61c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff620ff9700 1 -- 172.21.15.64:0/1730495917 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c00e040 con 0x7ff624100690 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62419db20 con 0x7ff624100690 2023-07-14T18:23:12.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff620ff9700 1 -- 172.21.15.64:0/1730495917 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff61c003e90 con 0x7ff624100690 2023-07-14T18:23:12.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.480+0000 7ff620ff9700 1 -- 172.21.15.64:0/1730495917 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c004020 con 0x7ff624100690 2023-07-14T18:23:12.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.481+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62419dff0 con 0x7ff624100690 2023-07-14T18:23:12.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.482+0000 7ff620ff9700 1 -- 172.21.15.64:0/1730495917 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff61c01d720 con 0x7ff624100690 2023-07-14T18:23:12.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.483+0000 7ff620ff9700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 0x7ff6100a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.543 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.483+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 0x7ff6100a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.544 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.483+0000 7ff620ff9700 1 -- 172.21.15.64:0/1730495917 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff61c0ce2e0 con 0x7ff624100690 2023-07-14T18:23:12.544 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 0x7ff6100a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff614005950 tx=0x7ff614005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.544 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 Processor -- start 2023-07-14T18:23:12.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 -- start start 2023-07-14T18:23:12.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bb860 0x55ec802bbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.545 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 0x55ec802bc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec801bfc60 con 0x55ec802baf10 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec802129f0 con 0x55ec802bc460 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec8020be80 con 0x55ec802bb860 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff6237fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff6237fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46988/0 (socket says 172.21.15.64:46988) 2023-07-14T18:23:12.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.484+0000 7ff6237fe700 1 -- 172.21.15.64:0/4140393523 learned_addr learned my addr 172.21.15.64:0/4140393523 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff623fff700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 0x55ec802bc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff622ffd700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bb860 0x55ec802bbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 -- 172.21.15.64:0/4140393523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bb860 msgr2=0x55ec802bbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bb860 0x55ec802bbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 -- 172.21.15.64:0/4140393523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 msgr2=0x55ec802bc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 0x55ec802bc8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 -- 172.21.15.64:0/4140393523 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec80039d90 con 0x55ec802baf10 2023-07-14T18:23:12.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff623fff700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 0x55ec802bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff6237fe700 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff61400ed10 tx=0x7ff614011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a74406daa92e836c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff619ffb700 1 -- 172.21.15.64:0/4140393523 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614010d80 con 0x55ec802baf10 2023-07-14T18:23:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff619ffb700 1 -- 172.21.15.64:0/4140393523 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff614004cc0 con 0x55ec802baf10 2023-07-14T18:23:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.485+0000 7ff619ffb700 1 -- 172.21.15.64:0/4140393523 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61401d450 con 0x55ec802baf10 2023-07-14T18:23:12.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- 172.21.15.64:0/4140393523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 msgr2=0x55ec802bb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff61400ed10 tx=0x7ff614011000 comp rx=0 tx=0).stop 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- 172.21.15.64:0/4140393523 shutdown_connections 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bc460 0x55ec802bc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802bb320 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/4140393523 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bb860 0x55ec802bbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- 172.21.15.64:0/4140393523 >> 172.21.15.64:0/4140393523 conn(0x55ec802b0630 msgr2=0x55ec802b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- 172.21.15.64:0/4140393523 shutdown_connections 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- 172.21.15.64:0/4140393523 wait complete. 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 -- start start 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.486+0000 7ff635ee7b80 1 Processor -- start 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802baf10 0x55ec802c4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec801aba70 con 0x55ec802bb860 2023-07-14T18:23:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec801ab770 con 0x55ec802baf10 2023-07-14T18:23:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec8008f8d0 con 0x55ec802bc460 2023-07-14T18:23:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 10 client.0 _ll_register_callbacks cb 0x55ec801a9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:23:12.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:46996/0 (socket says 172.21.15.64:46996) 2023-07-14T18:23:12.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 -- 172.21.15.64:0/1393685060 learned_addr learned my addr 172.21.15.64:0/1393685060 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802baf10 0x55ec802c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff623fff700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 -- 172.21.15.64:0/1393685060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 msgr2=0x55ec802cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802cbea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 -- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802baf10 msgr2=0x55ec802c4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802baf10 0x55ec802c4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 -- 172.21.15.64:0/1393685060 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61400e9c0 con 0x55ec802bb860 2023-07-14T18:23:12.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff623fff700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff622ffd700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff60c011260 tx=0x7ff60c011570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf13429c16c5985 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff602ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c015660 con 0x55ec802bb860 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff60c0157c0 con 0x55ec802bb860 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec801654a0 con 0x55ec802bb860 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.487+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60c011eb0 con 0x55ec802bb860 2023-07-14T18:23:12.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 10 client.14745 fetch_fsmap learned FSMap version 17 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55ec80022c10 con 0x55ec802bb860 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55ec801654a0 con 0x55ec802bb860 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ff60c014a90 con 0x55ec802bb860 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 10 client.14745 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 20 client.14745 populate_metadata read hostname 'smithi064' 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec802cfb70 0x55ec802cef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 4 client.14745 mds_command: new command op to 24545 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec80289d50 con 0x55ec802cfb70 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.488+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec802cfb70 0x55ec802cef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.489+0000 7ff6237fe700 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec802cfb70 0x55ec802cef20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ed5ad6701819f1e server_cookie=8c5d856e6e5278f7 in_seq=0 out_seq=0 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.489+0000 7ff602ffd700 10 client.14745 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.489+0000 7ff602ffd700 1 -- 172.21.15.64:0/1393685060 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55ec80289d50 con 0x55ec802cfb70 2023-07-14T18:23:12.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.489+0000 7ff602ffd700 10 client.14745 handle_command_reply: tid=0 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 client.14745 shutdown 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 msgr2=0x55ec802c4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff60c011260 tx=0x7ff60c011570 comp rx=0 tx=0).stop 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 shutdown_connections 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec802cfb70 0x55ec802cef20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802baf10 0x55ec802c4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/1393685060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bb860 0x55ec802c4e60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff602ffd700 0 client.14745 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:23:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 >> 172.21.15.64:0/1393685060 conn(0x55ec802b0630 msgr2=0x55ec802b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 shutdown_connections 2023-07-14T18:23:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 1 -- 172.21.15.64:0/1393685060 wait complete. 2023-07-14T18:23:12.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.490+0000 7ff635ee7b80 20 client.14745 trim_cache size 0 max 16384 2023-07-14T18:23:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 Processor -- start 2023-07-14T18:23:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 -- start start 2023-07-14T18:23:12.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 0x55ec802b50a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec80289d50 con 0x55ec802bc460 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec801654a0 con 0x55ec802bb860 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec801da790 con 0x55ec802baf10 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47006/0 (socket says 172.21.15.64:47006) 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 -- 172.21.15.64:0/3147404403 learned_addr learned my addr 172.21.15.64:0/3147404403 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff622ffd700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff6237fe700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 0x55ec802b50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 -- 172.21.15.64:0/3147404403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 msgr2=0x55ec802b50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 0x55ec802b50a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 -- 172.21.15.64:0/3147404403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 msgr2=0x55ec802cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 -- 172.21.15.64:0/3147404403 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec802b5f00 con 0x55ec802bc460 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff622ffd700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff6237fe700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 0x55ec802b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.508+0000 7ff623fff700 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff61c097a90 tx=0x7ff61c0bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61b47117940963c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff61a7fc700 1 -- 172.21.15.64:0/3147404403 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c09b590 con 0x55ec802bc460 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff61a7fc700 1 -- 172.21.15.64:0/3147404403 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff61c09f5c0 con 0x55ec802bc460 2023-07-14T18:23:12.563 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff61a7fc700 1 -- 172.21.15.64:0/3147404403 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c09b7f0 con 0x55ec802bc460 2023-07-14T18:23:12.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 -- 172.21.15.64:0/3147404403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 msgr2=0x55ec802c2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff61c097a90 tx=0x7ff61c0bb040 comp rx=0 tx=0).stop 2023-07-14T18:23:12.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 -- 172.21.15.64:0/3147404403 shutdown_connections 2023-07-14T18:23:12.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802baf10 0x55ec802b50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802bc460 0x55ec802c2840 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/3147404403 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 -- 172.21.15.64:0/3147404403 >> 172.21.15.64:0/3147404403 conn(0x55ec802b0630 msgr2=0x55ec802c3620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 -- 172.21.15.64:0/3147404403 shutdown_connections 2023-07-14T18:23:12.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.509+0000 7ff635ee7b80 1 -- 172.21.15.64:0/3147404403 wait complete. 2023-07-14T18:23:12.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 -- start start 2023-07-14T18:23:12.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 Processor -- start 2023-07-14T18:23:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802c7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802ca820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec802cad60 con 0x55ec802baf10 2023-07-14T18:23:12.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec802caed0 con 0x55ec802bb860 2023-07-14T18:23:12.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec802cb090 con 0x55ec802bc460 2023-07-14T18:23:12.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff635ee7b80 10 client.0 _ll_register_callbacks cb 0x7ff60c009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47008/0 (socket says 172.21.15.64:47008) 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 -- 172.21.15.64:0/968145726 learned_addr learned my addr 172.21.15.64:0/968145726 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff622ffd700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802c7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff623fff700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 -- 172.21.15.64:0/968145726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 msgr2=0x55ec802ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 -- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 msgr2=0x55ec802c7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:23:12.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802c7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.510+0000 7ff6237fe700 1 -- 172.21.15.64:0/968145726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62419dff0 con 0x55ec802baf10 2023-07-14T18:23:12.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff623fff700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:12.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff6237fe700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff61400ece0 tx=0x7ff614005240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d9da254d9c4f75 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:12.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:23:12.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6140053a0 con 0x55ec802baf10 2023-07-14T18:23:12.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff614010b80 con 0x55ec802baf10 2023-07-14T18:23:12.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff614004cc0 con 0x55ec802baf10 2023-07-14T18:23:12.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff614010d40 con 0x55ec802baf10 2023-07-14T18:23:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 10 client.14751 fetch_fsmap learned FSMap version 17 2023-07-14T18:23:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x55ec80060200 con 0x55ec802baf10 2023-07-14T18:23:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff614004cc0 con 0x55ec802baf10 2023-07-14T18:23:12.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7ff6140104e0 con 0x55ec802baf10 2023-07-14T18:23:12.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T18:23:12.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:23:12.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 20 client.14751 populate_metadata read hostname 'smithi064' 2023-07-14T18:23:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec803290f0 0x55ec80325a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 4 client.14751 mds_command: new command op to 24545 tid=0[{"prefix": "status"}] 2023-07-14T18:23:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.511+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec80204e60 con 0x55ec803290f0 2023-07-14T18:23:12.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff622ffd700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec803290f0 0x55ec80325a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff602ffd700 10 client.14751 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:23:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff622ffd700 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec803290f0 0x55ec80325a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed0d0e4db41b9031 server_cookie=5f7c44f5b13739a7 in_seq=0 out_seq=0 2023-07-14T18:23:12.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff602ffd700 1 -- 172.21.15.64:0/968145726 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55ec80204e60 con 0x55ec803290f0 2023-07-14T18:23:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff602ffd700 10 client.14751 handle_command_reply: tid=0 2023-07-14T18:23:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 client.14751 shutdown 2023-07-14T18:23:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 msgr2=0x55ec802c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff61400ece0 tx=0x7ff614005240 comp rx=0 tx=0).stop 2023-07-14T18:23:12.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 shutdown_connections 2023-07-14T18:23:12.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55ec803290f0 0x55ec80325a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55ec802bc460 0x55ec802ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff602ffd700 0 client.14751 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:23:12.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ec802baf10 0x55ec802c72c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 --2- 172.21.15.64:0/968145726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55ec802bb860 0x55ec802c7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.512+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 >> 172.21.15.64:0/968145726 conn(0x55ec802b0630 msgr2=0x55ec802b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 shutdown_connections 2023-07-14T18:23:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff635ee7b80 1 -- 172.21.15.64:0/968145726 wait complete. 2023-07-14T18:23:12.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff635ee7b80 20 client.14751 trim_cache size 0 max 16384 2023-07-14T18:23:12.585 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:23:12.585 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24545, 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:23:12.586 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 7.0975137970000004, 2023-07-14T18:23:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 17, 2023-07-14T18:23:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 85, 2023-07-14T18:23:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:23:12.587 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 13.20261694 2023-07-14T18:23:12.587 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:23:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 msgr2=0x7ff6100a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 0x7ff6100a3600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff614005950 tx=0x7ff614005f50 comp rx=0 tx=0).stop 2023-07-14T18:23:12.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 msgr2=0x7ff62419d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009fd0 tx=0x7ff61c004c80 comp rx=0 tx=0).stop 2023-07-14T18:23:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 shutdown_connections 2023-07-14T18:23:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff6240feb10 0x7ff624195a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff6100a1150 0x7ff6100a3600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff624100690 0x7ff62419d000 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 --2- 172.21.15.64:0/1730495917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6240ffd00 0x7ff624195f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:12.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 >> 172.21.15.64:0/1730495917 conn(0x7ff6240fa130 msgr2=0x7ff6240fa990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:12.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 shutdown_connections 2023-07-14T18:23:12.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:12.513+0000 7ff629d4f700 1 -- 172.21.15.64:0/1730495917 wait complete. 2023-07-14T18:23:12.591 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T18:23:12.592 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:12.593 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T18:23:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:12 smithi105 ceph-mon[98013]: pgmap v258: 129 pgs: 108 active+clean, 7 creating+peering, 14 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2023-07-14T18:23:12.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1851793612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3051199466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4072309881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:12 smithi064 ceph-mon[96720]: pgmap v258: 129 pgs: 108 active+clean, 7 creating+peering, 14 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2023-07-14T18:23:12.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1851793612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3051199466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4072309881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:12 smithi144 ceph-mon[98417]: pgmap v258: 129 pgs: 108 active+clean, 7 creating+peering, 14 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 4 op/s 2023-07-14T18:23:12.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1851793612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3051199466' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:12.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4072309881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:13.148 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:13.637 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:13 smithi064 ceph-mon[96720]: pgmap v259: 129 pgs: 122 active+clean, 7 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2023-07-14T18:23:13.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:13 smithi144 ceph-mon[98417]: pgmap v259: 129 pgs: 122 active+clean, 7 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2023-07-14T18:23:14.030 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:13 smithi105 ceph-mon[98013]: pgmap v259: 129 pgs: 122 active+clean, 7 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 3 op/s 2023-07-14T18:23:14.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 -- 172.21.15.64:0/2021437169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc105620 msgr2=0x7f67dc0ff4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:14.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/2021437169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc105620 0x7f67dc0ff4b0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67c40099c0 tx=0x7f67c4009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 -- 172.21.15.64:0/2021437169 shutdown_connections 2023-07-14T18:23:14.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/2021437169 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 0x7f67dc1050e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/2021437169 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc10ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/2021437169 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc105620 0x7f67dc0ff4b0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 -- 172.21.15.64:0/2021437169 >> 172.21.15.64:0/2021437169 conn(0x7f67dc076370 msgr2=0x7f67dc076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:14.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 -- 172.21.15.64:0/2021437169 shutdown_connections 2023-07-14T18:23:14.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.327+0000 7f67e0fa0700 1 -- 172.21.15.64:0/2021437169 wait complete. 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.328+0000 7f67e0fa0700 1 Processor -- start 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.328+0000 7f67e0fa0700 1 -- start start 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 0x7f67dc1971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc197860 0x7f67dc1a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:14.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc1a24b0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc1a2620 con 0x7f67dc197860 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67e0fa0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67dc1a27e0 con 0x7f67dc104d10 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47022/0 (socket says 172.21.15.64:47022) 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 -- 172.21.15.64:0/1767824895 learned_addr learned my addr 172.21.15.64:0/1767824895 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67d9d9b700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 0x7f67dc1971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:14.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67dad9d700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc197860 0x7f67dc1a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 -- 172.21.15.64:0/1767824895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 msgr2=0x7f67dc1971c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 0x7f67dc1971c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 -- 172.21.15.64:0/1767824895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc197860 msgr2=0x7f67dc1a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc197860 0x7f67dc1a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 -- 172.21.15.64:0/1767824895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c40095f0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.329+0000 7f67da59c700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00b530 tx=0x7f67cc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc011830 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f67cc011e20 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67dc1a2ac0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc00f2e0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67dc1a2f90 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.330+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67dc04e340 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.332+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f67cc011990 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.332+0000 7f67d37fe700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 0x7f67c80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:14.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.333+0000 7f67d9d9b700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 0x7f67c80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:14.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.333+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f67cc0cca40 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.333+0000 7f67d9d9b700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 0x7f67c80a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f67c40098e0 tx=0x7f67c4003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:14.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.336+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f67cc095fa0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.628 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.577+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f67dc1068a0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.577+0000 7f67d37fe700 1 -- 172.21.15.64:0/1767824895 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f67cc0995c0 con 0x7f67dc0ffbe0 2023-07-14T18:23:14.629 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 msgr2=0x7f67c80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 0x7f67c80a3600 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f67c40098e0 tx=0x7f67c4003680 comp rx=0 tx=0).stop 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 msgr2=0x7f67dc196c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67cc00b530 tx=0x7f67cc00b840 comp rx=0 tx=0).stop 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 shutdown_connections 2023-07-14T18:23:14.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67c80a1150 0x7f67c80a3600 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67dc197860 0x7f67dc1a1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67dc0ffbe0 0x7f67dc196c80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 --2- 172.21.15.64:0/1767824895 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67dc104d10 0x7f67dc1971c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 >> 172.21.15.64:0/1767824895 conn(0x7f67dc076370 msgr2=0x7f67dc1029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 shutdown_connections 2023-07-14T18:23:14.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:14.579+0000 7f67e0fa0700 1 -- 172.21.15.64:0/1767824895 wait complete. 2023-07-14T18:23:15.009 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-14T18:23:15.010 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T18:23:15.010 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2023-07-14T18:23:15.010 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-14T18:23:15.010 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:15.011 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-14T18:23:15.011 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T18:23:15.011 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T18:23:15.011 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T18:23:15.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.012 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2023-07-14T18:23:15.030 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T18:23:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T18:23:15.030 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:23:15.031 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 41671sec preferred_lft 41671sec 2023-07-14T18:23:15.032 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::bf40:81a3:b7e0:f244/64 scope link noprefixroute 2023-07-14T18:23:15.033 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:23:15.034 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T18:23:15.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.035 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:23:15.035 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:23:15.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2023-07-14T18:23:15.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2023-07-14T18:23:15.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2023-07-14T18:23:15.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T18:23:15.037 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:23:15.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.197 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T18:23:15.230 INFO:teuthology.orchestra.run.smithi064.stdout:1 2023-07-14T18:23:15.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.231 DEBUG:teuthology.orchestra.run.smithi064:> route 2023-07-14T18:23:15.311 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2023-07-14T18:23:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T18:23:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T18:23:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T18:23:15.312 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T18:23:15.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.313 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:23:15.314 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:23:15.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T18:23:15.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T18:23:15.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T18:23:15.315 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:23:15.502 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:15 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1767824895' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:23:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:15 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1767824895' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:23:15.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.584 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2023-07-14T18:23:15.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.602 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2023-07-14T18:23:15.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.660 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:23:15.660 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:23:15.660 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T18:23:15.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-14T18:23:15.661 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:23:15.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:15 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1767824895' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:23:15.759 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-14T18:23:15.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.760 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:23:15.760 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:23:15.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-14T18:23:15.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T18:23:15.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-14T18:23:15.761 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-14T18:23:15.762 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-14T18:23:15.762 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:23:15.969 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:15.969 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:23:15.970 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:23:15.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2023-07-14T18:23:15.970 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T18:23:15.970 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:23:16.072 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-14T18:23:16.073 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:23:16.073 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:16.091 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T18:23:16.132 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:23:16.132 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:16.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-07-14T18:23:16.218 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:23:16.218 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:23:16.286 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:23:16.288 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:23:16.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:16.289 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-07-14T18:23:16.347 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T18:23:16.347 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-14T18:23:16.390 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-07-14T18:23:16.419 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:23:16.419 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:23:16.486 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:23:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:23:16.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:16.489 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-07-14T18:23:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:2023-07-14T18:23:16.444+0000 7fb374fd35c0 -1 init, newargv = 0x563c830f8f50 newargc=15 2023-07-14T18:23:16.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[134022]: starting ceph client 2023-07-14T18:23:16.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[134022]: starting fuse 2023-07-14T18:23:16.514 INFO:teuthology.orchestra.run.smithi064.stdout:124 2023-07-14T18:23:16.514 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T18:23:16.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T18:23:16.515 DEBUG:teuthology.orchestra.run.smithi064:> import glob 2023-07-14T18:23:16.515 DEBUG:teuthology.orchestra.run.smithi064:> import re 2023-07-14T18:23:16.515 DEBUG:teuthology.orchestra.run.smithi064:> import os 2023-07-14T18:23:16.516 DEBUG:teuthology.orchestra.run.smithi064:> import subprocess 2023-07-14T18:23:16.516 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:23:16.516 DEBUG:teuthology.orchestra.run.smithi064:> def _find_admin_socket(client_name): 2023-07-14T18:23:16.517 DEBUG:teuthology.orchestra.run.smithi064:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-14T18:23:16.517 DEBUG:teuthology.orchestra.run.smithi064:> files = glob.glob(asok_path) 2023-07-14T18:23:16.517 DEBUG:teuthology.orchestra.run.smithi064:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-14T18:23:16.517 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:23:16.518 DEBUG:teuthology.orchestra.run.smithi064:> # Given a non-glob path, it better be there 2023-07-14T18:23:16.518 DEBUG:teuthology.orchestra.run.smithi064:> if "*" not in asok_path: 2023-07-14T18:23:16.518 DEBUG:teuthology.orchestra.run.smithi064:> assert(len(files) == 1) 2023-07-14T18:23:16.519 DEBUG:teuthology.orchestra.run.smithi064:> return files[0] 2023-07-14T18:23:16.519 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:23:16.519 DEBUG:teuthology.orchestra.run.smithi064:> for f in files: 2023-07-14T18:23:16.519 DEBUG:teuthology.orchestra.run.smithi064:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T18:23:16.520 DEBUG:teuthology.orchestra.run.smithi064:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T18:23:16.520 DEBUG:teuthology.orchestra.run.smithi064:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T18:23:16.520 DEBUG:teuthology.orchestra.run.smithi064:> contents = proc_f.read() 2023-07-14T18:23:16.520 DEBUG:teuthology.orchestra.run.smithi064:> if mountpoint in contents: 2023-07-14T18:23:16.521 DEBUG:teuthology.orchestra.run.smithi064:> return f 2023-07-14T18:23:16.521 DEBUG:teuthology.orchestra.run.smithi064:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T18:23:16.521 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:23:16.522 DEBUG:teuthology.orchestra.run.smithi064:> print(_find_admin_socket("client.admin")) 2023-07-14T18:23:16.522 DEBUG:teuthology.orchestra.run.smithi064:> ' 2023-07-14T18:23:16.560 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:16 smithi105 ceph-mon[98013]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:23:16.578 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:16 smithi064 ceph-mon[96720]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:23:16.640 INFO:teuthology.orchestra.run.smithi064.stdout:/var/run/ceph/ceph-client.admin.134022.asok 2023-07-14T18:23:16.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:16 smithi144 ceph-mon[98417]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2023-07-14T18:23:16.823 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.134022.asok 2023-07-14T18:23:16.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:16.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.134022.asok status 2023-07-14T18:23:16.987 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:23:16.988 INFO:teuthology.orchestra.run.smithi064.stdout: "metadata": { 2023-07-14T18:23:16.988 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T18:23:16.988 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:23:16.989 INFO:teuthology.orchestra.run.smithi064.stdout: "entity_id": "admin", 2023-07-14T18:23:16.989 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": "smithi064", 2023-07-14T18:23:16.989 INFO:teuthology.orchestra.run.smithi064.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-14T18:23:16.989 INFO:teuthology.orchestra.run.smithi064.stdout: "pid": "134022", 2023-07-14T18:23:16.990 INFO:teuthology.orchestra.run.smithi064.stdout: "root": "/" 2023-07-14T18:23:16.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:23:16.990 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_count": 0, 2023-07-14T18:23:16.990 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_pinned_count": 0, 2023-07-14T18:23:16.991 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 14763, 2023-07-14T18:23:16.991 INFO:teuthology.orchestra.run.smithi064.stdout: "inst": { 2023-07-14T18:23:16.991 INFO:teuthology.orchestra.run.smithi064.stdout: "name": { 2023-07-14T18:23:16.991 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "client", 2023-07-14T18:23:16.992 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 14763 2023-07-14T18:23:16.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:23:16.992 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-07-14T18:23:16.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-07-14T18:23:16.993 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:23:16.993 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2484308784 2023-07-14T18:23:16.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-07-14T18:23:16.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:23:16.994 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-07-14T18:23:16.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-07-14T18:23:16.994 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:23:16.995 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 2484308784 2023-07-14T18:23:16.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:23:16.995 INFO:teuthology.orchestra.run.smithi064.stdout: "inst_str": "client.14763 192.168.0.1:0/2484308784", 2023-07-14T18:23:16.995 INFO:teuthology.orchestra.run.smithi064.stdout: "addr_str": "192.168.0.1:0/2484308784", 2023-07-14T18:23:16.995 INFO:teuthology.orchestra.run.smithi064.stdout: "inode_count": 1, 2023-07-14T18:23:16.996 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_epoch": 17, 2023-07-14T18:23:16.996 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch": 85, 2023-07-14T18:23:16.996 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch_barrier": 0, 2023-07-14T18:23:16.996 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklisted": false, 2023-07-14T18:23:16.997 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs" 2023-07-14T18:23:16.997 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:23:16.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:16.999 DEBUG:teuthology.orchestra.run.smithi064:> stat --file-system '--printf=%T 2023-07-14T18:23:17.000 DEBUG:teuthology.orchestra.run.smithi064:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:17.018 INFO:teuthology.orchestra.run.smithi064.stdout:fuseblk 2023-07-14T18:23:17.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:17.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:17.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:17.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:17.088 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-14T18:23:17.193 INFO:teuthology.orchestra.run.smithi064.stdout:# file: . 2023-07-14T18:23:17.193 INFO:teuthology.orchestra.run.smithi064.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-14T18:23:17.193 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:17.194 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:23:17.194 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:23:17.211 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:17.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 -- 172.21.15.64:0/4033985457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 msgr2=0x7f240c100040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 --2- 172.21.15.64:0/4033985457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c100040 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f23fc009990 tx=0x7f23fc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 -- 172.21.15.64:0/4033985457 shutdown_connections 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 --2- 172.21.15.64:0/4033985457 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 --2- 172.21.15.64:0/4033985457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c100040 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 --2- 172.21.15.64:0/4033985457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 -- 172.21.15.64:0/4033985457 >> 172.21.15.64:0/4033985457 conn(0x7f240c0fb210 msgr2=0x7f240c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 -- 172.21.15.64:0/4033985457 shutdown_connections 2023-07-14T18:23:17.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.327+0000 7f2411fd8700 1 -- 172.21.15.64:0/4033985457 wait complete. 2023-07-14T18:23:17.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 Processor -- start 2023-07-14T18:23:17.382 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 -- start start 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c19d3e0 con 0x7f240c0ffc30 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c19d520 con 0x7f240c100e20 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f2411fd8700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240c19d6b0 con 0x7f240c1017b0 2023-07-14T18:23:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f240b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f240b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47046/0 (socket says 172.21.15.64:47046) 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.328+0000 7f240b7fe700 1 -- 172.21.15.64:0/1979380314 learned_addr learned my addr 172.21.15.64:0/1979380314 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240bfff700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240affd700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 -- 172.21.15.64:0/1979380314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 msgr2=0x7f240c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c19ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 -- 172.21.15.64:0/1979380314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 msgr2=0x7f240c195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 -- 172.21.15.64:0/1979380314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc0095f0 con 0x7f240c0ffc30 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240bfff700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240affd700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f240b7fe700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23fc000c00 tx=0x7f23fc009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc01d070 con 0x7f240c0ffc30 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f23fc0040b0 con 0x7f240c0ffc30 2023-07-14T18:23:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240c19d930 con 0x7f240c0ffc30 2023-07-14T18:23:17.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc022cb0 con 0x7f240c0ffc30 2023-07-14T18:23:17.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.329+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240c19dde0 con 0x7f240c0ffc30 2023-07-14T18:23:17.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.330+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240c106bb0 con 0x7f240c0ffc30 2023-07-14T18:23:17.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.332+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f23fc022470 con 0x7f240c0ffc30 2023-07-14T18:23:17.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.332+0000 7f2408ff9700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 0x7f23f40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.333+0000 7f240affd700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 0x7f23f40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.333+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f23fc0ce6d0 con 0x7f240c0ffc30 2023-07-14T18:23:17.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.333+0000 7f240affd700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 0x7f23f40a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2400005d90 tx=0x7f2400005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:17.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.336+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f23fc097cc0 con 0x7f240c0ffc30 2023-07-14T18:23:17.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.538+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f240c02c8e0 con 0x7f240c0ffc30 2023-07-14T18:23:17.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.539+0000 7f2408ff9700 1 -- 172.21.15.64:0/1979380314 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f23fc09b2e0 con 0x7f240c0ffc30 2023-07-14T18:23:17.591 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:17.593 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:17.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.541+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 msgr2=0x7f23f40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 0x7f23f40a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2400005d90 tx=0x7f2400005d20 comp rx=0 tx=0).stop 2023-07-14T18:23:17.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 msgr2=0x7f240c195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f23fc000c00 tx=0x7f23fc009f90 comp rx=0 tx=0).stop 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 shutdown_connections 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f240c1017b0 0x7f240c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f40a11a0 0x7f23f40a3650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f240c0ffc30 0x7f240c195860 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 --2- 172.21.15.64:0/1979380314 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f240c100e20 0x7f240c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 >> 172.21.15.64:0/1979380314 conn(0x7f240c0fb210 msgr2=0x7f240c0fd520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:17.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 shutdown_connections 2023-07-14T18:23:17.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.542+0000 7f2411fd8700 1 -- 172.21.15.64:0/1979380314 wait complete. 2023-07-14T18:23:17.606 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:23:17.608 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:23:17.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2594185197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c100690 msgr2=0x7f9c9c106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2594185197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c100690 0x7f9c9c106f50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9c9400b0a0 tx=0x7f9c9400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2594185197 shutdown_connections 2023-07-14T18:23:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2594185197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c100690 0x7f9c9c106f50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2594185197 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2594185197 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2594185197 >> 172.21.15.64:0/2594185197 conn(0x7f9c9c0fa130 msgr2=0x7f9c9c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:17.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2594185197 shutdown_connections 2023-07-14T18:23:17.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.686+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2594185197 wait complete. 2023-07-14T18:23:17.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 Processor -- start 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 -- start start 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c1010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 0x7f9c9c101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c101dc0 con 0x7f9c9c0feb10 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c101f30 con 0x7f9c9c100690 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca3c0f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9c19f9b0 con 0x7f9c9c0ffd00 2023-07-14T18:23:17.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47060/0 (socket says 172.21.15.64:47060) 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 -- 172.21.15.64:0/2172959938 learned_addr learned my addr 172.21.15.64:0/2172959938 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca21ac700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 0x7f9c9c101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca11aa700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c1010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 -- 172.21.15.64:0/2172959938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 msgr2=0x7f9c9c1010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c1010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 -- 172.21.15.64:0/2172959938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 msgr2=0x7f9c9c101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 0x7f9c9c101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca19ab700 1 -- 172.21.15.64:0/2172959938 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94009d00 con 0x7f9c9c0feb10 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca21ac700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 0x7f9c9c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.687+0000 7f9ca11aa700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9ca19ab700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800d980 tx=0x7f9c9800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9800de60 con 0x7f9c9c0feb10 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9c980044b0 con 0x7f9c9c0feb10 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9c19fc40 con 0x7f9c9c0feb10 2023-07-14T18:23:17.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98003aa0 con 0x7f9c9c0feb10 2023-07-14T18:23:17.746 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.688+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9c1a01a0 con 0x7f9c9c0feb10 2023-07-14T18:23:17.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.689+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9c10a1a0 con 0x7f9c9c0feb10 2023-07-14T18:23:17.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.690+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9c98004870 con 0x7f9c9c0feb10 2023-07-14T18:23:17.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.690+0000 7f9c8effd700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 0x7f9c880a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:17.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.691+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9c980ccb30 con 0x7f9c9c0feb10 2023-07-14T18:23:17.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.691+0000 7f9ca11aa700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 0x7f9c880a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:17.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.691+0000 7f9ca11aa700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 0x7f9c880a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005fd0 tx=0x7f9c90005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:17.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.694+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c98096160 con 0x7f9c9c0feb10 2023-07-14T18:23:17.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:17 smithi064 ceph-mon[96720]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 382 B/s wr, 0 op/s 2023-07-14T18:23:17.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:17 smithi144 ceph-mon[98417]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 382 B/s wr, 0 op/s 2023-07-14T18:23:17.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.892+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9c9c1a0750 con 0x7f9c9c0feb10 2023-07-14T18:23:17.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.892+0000 7f9c8effd700 1 -- 172.21.15.64:0/2172959938 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9c98017020 con 0x7f9c9c0feb10 2023-07-14T18:23:17.944 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:23:17.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 msgr2=0x7f9c880a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 0x7f9c880a3680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c90005fd0 tx=0x7f9c90005e20 comp rx=0 tx=0).stop 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 msgr2=0x7f9c9c102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800d980 tx=0x7f9c9800dd40 comp rx=0 tx=0).stop 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 shutdown_connections 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f9c880a11d0 0x7f9c880a3680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9c9c100690 0x7f9c9c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c9c0feb10 0x7f9c9c102a50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 --2- 172.21.15.64:0/2172959938 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9c9c0ffd00 0x7f9c9c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:17.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 >> 172.21.15.64:0/2172959938 conn(0x7f9c9c0fa130 msgr2=0x7f9c9c0677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:17.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 shutdown_connections 2023-07-14T18:23:17.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:17.895+0000 7f9ca3c0f700 1 -- 172.21.15.64:0/2172959938 wait complete. 2023-07-14T18:23:17.958 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:23:18.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:17 smithi105 ceph-mon[98013]: pgmap v261: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 382 B/s wr, 0 op/s 2023-07-14T18:23:18.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1390751423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 msgr2=0x7fcd200fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1390751423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd200fef50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009990 tx=0x7fcd1c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1390751423 shutdown_connections 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1390751423 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd201006c0 0x7fcd20106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1390751423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd200fef50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1390751423 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd200ffd30 0x7fcd20100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1390751423 >> 172.21.15.64:0/1390751423 conn(0x7fcd200fa160 msgr2=0x7fcd200fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:18.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1390751423 shutdown_connections 2023-07-14T18:23:18.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1390751423 wait complete. 2023-07-14T18:23:18.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.036+0000 7fcd27cf5700 1 Processor -- start 2023-07-14T18:23:18.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 -- start start 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd200ffd30 0x7fcd2019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd201006c0 0x7fcd2019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2019d420 con 0x7fcd200feb40 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2019d590 con 0x7fcd200ffd30 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd27cf5700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd2019d750 con 0x7fcd201006c0 2023-07-14T18:23:18.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47074/0 (socket says 172.21.15.64:47074) 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 -- 172.21.15.64:0/1149943172 learned_addr learned my addr 172.21.15.64:0/1149943172 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25290700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd200ffd30 0x7fcd2019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 -- 172.21.15.64:0/1149943172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd201006c0 msgr2=0x7fcd2019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd201006c0 0x7fcd2019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 -- 172.21.15.64:0/1149943172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd200ffd30 msgr2=0x7fcd2019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd200ffd30 0x7fcd2019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 -- 172.21.15.64:0/1149943172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c0095f0 con 0x7fcd200feb40 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.037+0000 7fcd25a91700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c000c00 tx=0x7fcd1c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.038+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c01d070 con 0x7fcd200feb40 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.038+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd2019d9d0 con 0x7fcd200feb40 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.038+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcd1c0040b0 con 0x7fcd200feb40 2023-07-14T18:23:18.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.038+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c022cb0 con 0x7fcd200feb40 2023-07-14T18:23:18.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.038+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2019de80 con 0x7fcd200feb40 2023-07-14T18:23:18.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.039+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd2010a1c0 con 0x7fcd200feb40 2023-07-14T18:23:18.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.040+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcd1c022470 con 0x7fcd200feb40 2023-07-14T18:23:18.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.040+0000 7fcd16ffd700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 0x7fcd0c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.040+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcd1c0ce000 con 0x7fcd200feb40 2023-07-14T18:23:18.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.041+0000 7fcd25290700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 0x7fcd0c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.041+0000 7fcd25290700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 0x7fcd0c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd10005950 tx=0x7fcd100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.045+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcd1c0975f0 con 0x7fcd200feb40 2023-07-14T18:23:18.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.244+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd20105ac0 con 0x7fcd200feb40 2023-07-14T18:23:18.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.244+0000 7fcd16ffd700 1 -- 172.21.15.64:0/1149943172 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcd1c09ac10 con 0x7fcd200feb40 2023-07-14T18:23:18.296 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:23:18.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 msgr2=0x7fcd0c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 0x7fcd0c0a3600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcd10005950 tx=0x7fcd100058e0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 msgr2=0x7fcd20069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c000c00 tx=0x7fcd1c009f90 comp rx=0 tx=0).stop 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 shutdown_connections 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcd201006c0 0x7fcd2019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcd0c0a1150 0x7fcd0c0a3600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd200feb40 0x7fcd20069300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 --2- 172.21.15.64:0/1149943172 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd200ffd30 0x7fcd2019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 >> 172.21.15.64:0/1149943172 conn(0x7fcd200fa160 msgr2=0x7fcd200fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:18.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 shutdown_connections 2023-07-14T18:23:18.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.246+0000 7fcd27cf5700 1 -- 172.21.15.64:0/1149943172 wait complete. 2023-07-14T18:23:18.310 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:23:18.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 -- 172.21.15.64:0/3532429898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd00feb10 msgr2=0x7f1bd00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 --2- 172.21.15.64:0/3532429898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd00feb10 0x7f1bd00fef20 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009990 tx=0x7f1bc4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 -- 172.21.15.64:0/3532429898 shutdown_connections 2023-07-14T18:23:18.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 --2- 172.21.15.64:0/3532429898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd0100690 0x7f1bd0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 --2- 172.21.15.64:0/3532429898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd00feb10 0x7f1bd00fef20 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.387+0000 7f1bd7050700 1 --2- 172.21.15.64:0/3532429898 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00ffd00 0x7f1bd0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- 172.21.15.64:0/3532429898 >> 172.21.15.64:0/3532429898 conn(0x7f1bd00fa130 msgr2=0x7f1bd00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:18.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- 172.21.15.64:0/3532429898 shutdown_connections 2023-07-14T18:23:18.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- 172.21.15.64:0/3532429898 wait complete. 2023-07-14T18:23:18.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 Processor -- start 2023-07-14T18:23:18.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- start start 2023-07-14T18:23:18.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 0x7f1bd0102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 0x7f1bd01010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0101dc0 con 0x7f1bd0100690 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd0101f30 con 0x7f1bd00ffd00 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.388+0000 7f1bd7050700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bd019f950 con 0x7f1bd00feb10 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47092/0 (socket says 172.21.15.64:47092) 2023-07-14T18:23:18.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 -- 172.21.15.64:0/2024738621 learned_addr learned my addr 172.21.15.64:0/2024738621 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:18.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bcffff700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 0x7f1bd01010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd4dec700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 0x7f1bd0102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 -- 172.21.15.64:0/2024738621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 msgr2=0x7f1bd0102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 0x7f1bd0102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 -- 172.21.15.64:0/2024738621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 msgr2=0x7f1bd01010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 0x7f1bd01010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 -- 172.21.15.64:0/2024738621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40095f0 con 0x7f1bd0100690 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd4dec700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 0x7f1bd0102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:18.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bcffff700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 0x7f1bd01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd55ed700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800ada0 tx=0x7f1bc800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc800f070 con 0x7f1bd0100690 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1bc8007b30 con 0x7f1bd0100690 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bd019fc30 con 0x7f1bd0100690 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80053e0 con 0x7f1bd0100690 2023-07-14T18:23:18.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.389+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bd01a0140 con 0x7f1bd0100690 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.390+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bd010a1e0 con 0x7f1bd0100690 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.391+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1bc800c980 con 0x7f1bd0100690 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.392+0000 7f1bcdffb700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 0x7f1bc00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.392+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1bc80cc210 con 0x7f1bd0100690 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.392+0000 7f1bd4dec700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 0x7f1bc00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.392+0000 7f1bd4dec700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 0x7f1bc00a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400b5c0 tx=0x7f1bc4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.396+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1bc8095800 con 0x7f1bd0100690 2023-07-14T18:23:18.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.596+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1bd01a06f0 con 0x7f1bd0100690 2023-07-14T18:23:18.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.596+0000 7f1bcdffb700 1 -- 172.21.15.64:0/2024738621 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1bc800cc30 con 0x7f1bd0100690 2023-07-14T18:23:18.648 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:23:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 msgr2=0x7f1bc00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 0x7f1bc00a3600 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1bc400b5c0 tx=0x7f1bc4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 msgr2=0x7f1bd0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800ada0 tx=0x7f1bc800c330 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 shutdown_connections 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.598+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1bd00feb10 0x7f1bd0102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1bc00a1150 0x7f1bc00a3600 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1bd0100690 0x7f1bd0101620 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 --2- 172.21.15.64:0/2024738621 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1bd00ffd00 0x7f1bd01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.651 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 >> 172.21.15.64:0/2024738621 conn(0x7f1bd00fa130 msgr2=0x7f1bd00faa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 shutdown_connections 2023-07-14T18:23:18.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.599+0000 7f1bd7050700 1 -- 172.21.15.64:0/2024738621 wait complete. 2023-07-14T18:23:18.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:23:18.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 -- 172.21.15.64:0/1202641570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f481058d0 msgr2=0x7f0f48107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/1202641570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f481058d0 0x7f0f48107d30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000b0a0 tx=0x7f0f4000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 -- 172.21.15.64:0/1202641570 shutdown_connections 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/1202641570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f481058d0 0x7f0f48107d30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/1202641570 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48100690 0x7f0f48102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/1202641570 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f48102fb0 0x7f0f48105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 -- 172.21.15.64:0/1202641570 >> 172.21.15.64:0/1202641570 conn(0x7f0f480fa130 msgr2=0x7f0f480fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 -- 172.21.15.64:0/1202641570 shutdown_connections 2023-07-14T18:23:18.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.739+0000 7f0f4f11b700 1 -- 172.21.15.64:0/1202641570 wait complete. 2023-07-14T18:23:18.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 Processor -- start 2023-07-14T18:23:18.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 -- start start 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48102fb0 0x7f0f48195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f481058d0 0x7f0f4819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819d480 con 0x7f0f48100690 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819d5c0 con 0x7f0f48102fb0 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4f11b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819d750 con 0x7f0f481058d0 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4ceb7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4ceb7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47100/0 (socket says 172.21.15.64:47100) 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f4ceb7700 1 -- 172.21.15.64:0/3260441875 learned_addr learned my addr 172.21.15.64:0/3260441875 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.740+0000 7f0f47fff700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48102fb0 0x7f0f48195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4d6b8700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f481058d0 0x7f0f4819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 -- 172.21.15.64:0/3260441875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f481058d0 msgr2=0x7f0f4819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f481058d0 0x7f0f4819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 -- 172.21.15.64:0/3260441875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48102fb0 msgr2=0x7f0f48195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:18.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48102fb0 0x7f0f48195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 -- 172.21.15.64:0/3260441875 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f40009d00 con 0x7f0f48100690 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4ceb7700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00c920 tx=0x7f0f3c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c007b00 con 0x7f0f48100690 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f3c00ce70 con 0x7f0f48100690 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f3c005440 con 0x7f0f48100690 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f4819da30 con 0x7f0f48100690 2023-07-14T18:23:18.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.741+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f4819df40 con 0x7f0f48100690 2023-07-14T18:23:18.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.742+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f480fbca0 con 0x7f0f48100690 2023-07-14T18:23:18.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.743+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f3c0055a0 con 0x7f0f48100690 2023-07-14T18:23:18.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.744+0000 7f0f45ffb700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:18.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.744+0000 7f0f47fff700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:18.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.744+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0f3c0cc650 con 0x7f0f48100690 2023-07-14T18:23:18.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.744+0000 7f0f47fff700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400ac50 tx=0x7f0f3400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:18.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.748+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f3c095bb0 con 0x7f0f48100690 2023-07-14T18:23:18.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:18 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1979380314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:18.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:18 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2172959938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:18.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:18 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1149943172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:18.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:18 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1979380314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:18.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:18 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2172959938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:18.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:18 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1149943172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:18.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.947+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0f4802cd30 con 0x7f0f48100690 2023-07-14T18:23:19.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.948+0000 7f0f45ffb700 1 -- 172.21.15.64:0/3260441875 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0f3c0991d0 con 0x7f0f48100690 2023-07-14T18:23:19.000 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:23:19.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 msgr2=0x7f0f380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f0f3400ac50 tx=0x7f0f3400a380 comp rx=0 tx=0).stop 2023-07-14T18:23:19.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 msgr2=0x7f0f48195900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0f3c00c920 tx=0x7f0f3c00cc30 comp rx=0 tx=0).stop 2023-07-14T18:23:19.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 shutdown_connections 2023-07-14T18:23:19.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f481058d0 0x7f0f4819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48195900 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 --2- 172.21.15.64:0/3260441875 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f48102fb0 0x7f0f48195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 >> 172.21.15.64:0/3260441875 conn(0x7f0f480fa130 msgr2=0x7f0f48101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 shutdown_connections 2023-07-14T18:23:19.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:18.950+0000 7f0f4f11b700 1 -- 172.21.15.64:0/3260441875 wait complete. 2023-07-14T18:23:19.014 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:23:19.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:18 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1979380314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:19.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:18 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2172959938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:18 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1149943172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- 172.21.15.64:0/1963709803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 msgr2=0x7f415c0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 --2- 172.21.15.64:0/1963709803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c0fef20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f414c009990 tx=0x7f414c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- 172.21.15.64:0/1963709803 shutdown_connections 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 --2- 172.21.15.64:0/1963709803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c100690 0x7f415c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 --2- 172.21.15.64:0/1963709803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c0fef20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 --2- 172.21.15.64:0/1963709803 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c0ffd00 0x7f415c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- 172.21.15.64:0/1963709803 >> 172.21.15.64:0/1963709803 conn(0x7f415c0fa130 msgr2=0x7f415c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- 172.21.15.64:0/1963709803 shutdown_connections 2023-07-14T18:23:19.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- 172.21.15.64:0/1963709803 wait complete. 2023-07-14T18:23:19.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 Processor -- start 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.090+0000 7f4162978700 1 -- start start 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c0ffd00 0x7f415c107a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 0x7f415c107f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c108720 con 0x7f415c0feb10 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c108890 con 0x7f415c0ffd00 2023-07-14T18:23:19.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415c19f980 con 0x7f415c100690 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47112/0 (socket says 172.21.15.64:47112) 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 -- 172.21.15.64:0/2371086031 learned_addr learned my addr 172.21.15.64:0/2371086031 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415b7fe700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c0ffd00 0x7f415c107a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4160f15700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 0x7f415c107f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 -- 172.21.15.64:0/2371086031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 msgr2=0x7f415c107f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 0x7f415c107f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 -- 172.21.15.64:0/2371086031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c0ffd00 msgr2=0x7f415c107a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c0ffd00 0x7f415c107a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 -- 172.21.15.64:0/2371086031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c0095f0 con 0x7f415c0feb10 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4160f15700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 0x7f415c107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f415bfff700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098e0 tx=0x7f414c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c01d070 con 0x7f415c0feb10 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f414c00bdd0 con 0x7f415c0feb10 2023-07-14T18:23:19.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c19fc00 con 0x7f415c0feb10 2023-07-14T18:23:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c022d80 con 0x7f415c0feb10 2023-07-14T18:23:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.091+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c1a0020 con 0x7f415c0feb10 2023-07-14T18:23:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.092+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c105b80 con 0x7f415c0feb10 2023-07-14T18:23:19.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.093+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f414c022470 con 0x7f415c0feb10 2023-07-14T18:23:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.094+0000 7f41597fa700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 0x7f41440a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.094+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f414c0cdfd0 con 0x7f415c0feb10 2023-07-14T18:23:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.094+0000 7f415b7fe700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 0x7f41440a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.095+0000 7f415b7fe700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 0x7f41440a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4150009c60 tx=0x7f4150009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.098+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f414c0975c0 con 0x7f415c0feb10 2023-07-14T18:23:19.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.313+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f415c02cd00 con 0x7f415c0feb10 2023-07-14T18:23:19.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.313+0000 7f41597fa700 1 -- 172.21.15.64:0/2371086031 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+589 (secure 0 0 0) 0x7f414c09abe0 con 0x7f415c0feb10 2023-07-14T18:23:19.365 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2932389061 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3783129146 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1024618281 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:19.366 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/377975996 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:19.367 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1532744554 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:19.367 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2890765683 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:19.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 msgr2=0x7f41440a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 0x7f41440a3600 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4150009c60 tx=0x7f4150009260 comp rx=0 tx=0).stop 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 msgr2=0x7f415c109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f414c0098e0 tx=0x7f414c004730 comp rx=0 tx=0).stop 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 shutdown_connections 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f415c100690 0x7f415c107f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f41440a1150 0x7f41440a3600 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f415c0feb10 0x7f415c109400 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 --2- 172.21.15.64:0/2371086031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f415c0ffd00 0x7f415c107a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.315+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 >> 172.21.15.64:0/2371086031 conn(0x7f415c0fa130 msgr2=0x7f415c0fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.316+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 shutdown_connections 2023-07-14T18:23:19.371 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.316+0000 7f4162978700 1 -- 172.21.15.64:0/2371086031 wait complete. 2023-07-14T18:23:19.372 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2023-07-14T18:23:19.379 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:23:19.380 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:23:19.398 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:19.564 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 -- 172.21.15.64:0/2679929459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900ffd40 msgr2=0x7f5790100190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 --2- 172.21.15.64:0/2679929459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900ffd40 0x7f5790100190 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5778009990 tx=0x7f5778009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 -- 172.21.15.64:0/2679929459 shutdown_connections 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 --2- 172.21.15.64:0/2679929459 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57901006d0 0x7f5790106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 --2- 172.21.15.64:0/2679929459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900feb50 0x7f57900fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 --2- 172.21.15.64:0/2679929459 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900ffd40 0x7f5790100190 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.512+0000 7f5796a0c700 1 -- 172.21.15.64:0/2679929459 >> 172.21.15.64:0/2679929459 conn(0x7f57900fa190 msgr2=0x7f57900fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.513+0000 7f5796a0c700 1 -- 172.21.15.64:0/2679929459 shutdown_connections 2023-07-14T18:23:19.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.513+0000 7f5796a0c700 1 -- 172.21.15.64:0/2679929459 wait complete. 2023-07-14T18:23:19.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.513+0000 7f5796a0c700 1 Processor -- start 2023-07-14T18:23:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.513+0000 7f5796a0c700 1 -- start start 2023-07-14T18:23:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57900ffd40 0x7f579019a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 0x7f579019cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579019d360 con 0x7f57900feb50 2023-07-14T18:23:19.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579019d4d0 con 0x7f57901006d0 2023-07-14T18:23:19.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579019d690 con 0x7f57900ffd40 2023-07-14T18:23:19.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47126/0 (socket says 172.21.15.64:47126) 2023-07-14T18:23:19.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 -- 172.21.15.64:0/34354472 learned_addr learned my addr 172.21.15.64:0/34354472 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:19.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5787fff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57900ffd40 0x7f579019a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5794fa9700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 0x7f579019cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 -- 172.21.15.64:0/34354472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57900ffd40 msgr2=0x7f579019a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57900ffd40 0x7f579019a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 -- 172.21.15.64:0/34354472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 msgr2=0x7f579019cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 0x7f579019cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 -- 172.21.15.64:0/34354472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57780095f0 con 0x7f57900feb50 2023-07-14T18:23:19.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5794fa9700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 0x7f579019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:19.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578ffff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f578000e970 tx=0x7f578000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780009790 con 0x7f57900feb50 2023-07-14T18:23:19.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579019d970 con 0x7f57900feb50 2023-07-14T18:23:19.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f578000eec0 con 0x7f57900feb50 2023-07-14T18:23:19.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.514+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5780003c10 con 0x7f57900feb50 2023-07-14T18:23:19.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.515+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579019de80 con 0x7f57900feb50 2023-07-14T18:23:19.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.515+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57901060c0 con 0x7f57900feb50 2023-07-14T18:23:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.517+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5780003d90 con 0x7f57900feb50 2023-07-14T18:23:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.517+0000 7f578dffb700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 0x7f577c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.517+0000 7f5787fff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 0x7f577c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.517+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5780014070 con 0x7f57900feb50 2023-07-14T18:23:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.518+0000 7f5787fff700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 0x7f577c0a3650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5778006010 tx=0x7f577800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.521+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5780096440 con 0x7f57900feb50 2023-07-14T18:23:19.774 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.723+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f579002cab0 con 0x7f57900feb50 2023-07-14T18:23:19.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.724+0000 7f578dffb700 1 -- 172.21.15.64:0/34354472 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f5780099a60 con 0x7f57900feb50 2023-07-14T18:23:19.776 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:19.777 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:19.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 msgr2=0x7f577c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 0x7f577c0a3650 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5778006010 tx=0x7f577800b260 comp rx=0 tx=0).stop 2023-07-14T18:23:19.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 msgr2=0x7f57900692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f578000e970 tx=0x7f578000ec80 comp rx=0 tx=0).stop 2023-07-14T18:23:19.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 shutdown_connections 2023-07-14T18:23:19.782 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f577c0a11a0 0x7f577c0a3650 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.782 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57901006d0 0x7f579019cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.782 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57900feb50 0x7f57900692c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.782 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 --2- 172.21.15.64:0/34354472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57900ffd40 0x7f579019a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 >> 172.21.15.64:0/34354472 conn(0x7f57900fa190 msgr2=0x7f5790103200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 shutdown_connections 2023-07-14T18:23:19.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.727+0000 7f5796a0c700 1 -- 172.21.15.64:0/34354472 wait complete. 2023-07-14T18:23:19.789 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:19 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2024738621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:19 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3260441875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:19 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2371086031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:19.790 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:19 smithi064 ceph-mon[96720]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2023-07-14T18:23:19.791 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:23:19.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:23:19.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 -- 172.21.15.64:0/67446386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc100650 msgr2=0x7f1ddc106f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 --2- 172.21.15.64:0/67446386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc100650 0x7f1ddc106f10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1dd400b0a0 tx=0x7f1dd400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 -- 172.21.15.64:0/67446386 shutdown_connections 2023-07-14T18:23:19.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 --2- 172.21.15.64:0/67446386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc100650 0x7f1ddc106f10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 --2- 172.21.15.64:0/67446386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc0fead0 0x7f1ddc0feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 --2- 172.21.15.64:0/67446386 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc100110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.869+0000 7f1de335f700 1 -- 172.21.15.64:0/67446386 >> 172.21.15.64:0/67446386 conn(0x7f1ddc0fa0d0 msgr2=0x7f1ddc0fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:19.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.870+0000 7f1de335f700 1 -- 172.21.15.64:0/67446386 shutdown_connections 2023-07-14T18:23:19.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.870+0000 7f1de335f700 1 -- 172.21.15.64:0/67446386 wait complete. 2023-07-14T18:23:19.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.870+0000 7f1de335f700 1 Processor -- start 2023-07-14T18:23:19.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.870+0000 7f1de335f700 1 -- start start 2023-07-14T18:23:19.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0fead0 0x7f1ddc195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc100650 0x7f1ddc19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc19d4b0 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc19d5f0 con 0x7f1ddc100650 2023-07-14T18:23:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc19d780 con 0x7f1ddc0fead0 2023-07-14T18:23:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47142/0 (socket says 172.21.15.64:47142) 2023-07-14T18:23:19.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 -- 172.21.15.64:0/3057958049 learned_addr learned my addr 172.21.15.64:0/3057958049 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:19.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de10fb700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0fead0 0x7f1ddc195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de18fc700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc100650 0x7f1ddc19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 -- 172.21.15.64:0/3057958049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0fead0 msgr2=0x7f1ddc195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0fead0 0x7f1ddc195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 -- 172.21.15.64:0/3057958049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc100650 msgr2=0x7f1ddc19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc100650 0x7f1ddc19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 -- 172.21.15.64:0/3057958049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4009d00 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de08fa700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800d950 tx=0x7f1dd800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd800f890 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1dd800fe80 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd800e3b0 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.871+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc19da60 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.872+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc19df70 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.931 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.872+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc10a160 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.873+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1dd8010460 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.874+0000 7f1dce7fc700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 0x7f1dc80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:19.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.874+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1dd80cc860 con 0x7f1ddc0ffcc0 2023-07-14T18:23:19.932 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.874+0000 7f1de10fb700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 0x7f1dc80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:19.933 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.875+0000 7f1de10fb700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 0x7f1dc80a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0ffb20 tx=0x7f1dd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:19.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:19 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2024738621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:19 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3260441875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:19.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:19 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2371086031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:19.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:19 smithi144 ceph-mon[98417]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2023-07-14T18:23:19.934 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:19.879+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1dd8095dc0 con 0x7f1ddc0ffcc0 2023-07-14T18:23:20.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:19 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2024738621' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:19 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3260441875' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:19 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2371086031' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:20.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:19 smithi105 ceph-mon[98013]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2023-07-14T18:23:20.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.081+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1ddc19e700 con 0x7f1ddc0ffcc0 2023-07-14T18:23:20.134 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.082+0000 7f1dce7fc700 1 -- 172.21.15.64:0/3057958049 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1dd80160a0 con 0x7f1ddc0ffcc0 2023-07-14T18:23:20.134 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:23:20.135 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 msgr2=0x7f1dc80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 0x7f1dc80a3680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc0ffb20 tx=0x7f1dd000b410 comp rx=0 tx=0).stop 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 msgr2=0x7f1ddc195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800d950 tx=0x7f1dd800dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 shutdown_connections 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1ddc0fead0 0x7f1ddc195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.136 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1dc80a11d0 0x7f1dc80a3680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ddc100650 0x7f1ddc19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 --2- 172.21.15.64:0/3057958049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ddc0ffcc0 0x7f1ddc195e70 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 >> 172.21.15.64:0/3057958049 conn(0x7f1ddc0fa0d0 msgr2=0x7f1ddc0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 shutdown_connections 2023-07-14T18:23:20.137 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.084+0000 7f1de335f700 1 -- 172.21.15.64:0/3057958049 wait complete. 2023-07-14T18:23:20.148 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- 172.21.15.64:0/1431620364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 msgr2=0x7f2cd0072ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/1431620364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd0072ad0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009990 tx=0x7f2ccc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- 172.21.15.64:0/1431620364 shutdown_connections 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/1431620364 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 0x7f2cd010e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/1431620364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd0072ad0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/1431620364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd0073510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- 172.21.15.64:0/1431620364 >> 172.21.15.64:0/1431620364 conn(0x7f2cd00fb250 msgr2=0x7f2cd00fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- 172.21.15.64:0/1431620364 shutdown_connections 2023-07-14T18:23:20.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- 172.21.15.64:0/1431620364 wait complete. 2023-07-14T18:23:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 Processor -- start 2023-07-14T18:23:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.225+0000 7f2cd81d8700 1 -- start start 2023-07-14T18:23:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 0x7f2cd01a3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd01a3ae0 con 0x7f2cd0074680 2023-07-14T18:23:20.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd01a3c20 con 0x7f2cd00730a0 2023-07-14T18:23:20.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd81d8700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd01a3db0 con 0x7f2cd010c1e0 2023-07-14T18:23:20.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47156/0 (socket says 172.21.15.64:47156) 2023-07-14T18:23:20.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 -- 172.21.15.64:0/4064204708 learned_addr learned my addr 172.21.15.64:0/4064204708 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:20.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5f74700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd6775700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 0x7f2cd01a3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 -- 172.21.15.64:0/4064204708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 msgr2=0x7f2cd01a3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 0x7f2cd01a3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 -- 172.21.15.64:0/4064204708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 msgr2=0x7f2cd019bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd019bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 -- 172.21.15.64:0/4064204708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ccc0095f0 con 0x7f2cd0074680 2023-07-14T18:23:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5f74700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:20.285 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.226+0000 7f2cd5773700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000e970 tx=0x7f2cc000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.227+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0009790 con 0x7f2cd0074680 2023-07-14T18:23:20.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.227+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2cc00044b0 con 0x7f2cd0074680 2023-07-14T18:23:20.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.227+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd01a4090 con 0x7f2cd0074680 2023-07-14T18:23:20.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.227+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0003dd0 con 0x7f2cd0074680 2023-07-14T18:23:20.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.227+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd01a45f0 con 0x7f2cd0074680 2023-07-14T18:23:20.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.228+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd0101d30 con 0x7f2cd0074680 2023-07-14T18:23:20.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.229+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2cc0004000 con 0x7f2cd0074680 2023-07-14T18:23:20.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.230+0000 7f2cc6ffd700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 0x7f2cbc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.230+0000 7f2cd5f74700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 0x7f2cbc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.230+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2cc0014070 con 0x7f2cd0074680 2023-07-14T18:23:20.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.230+0000 7f2cd5f74700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 0x7f2cbc0a3680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098e0 tx=0x7f2ccc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.235+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2cc00963f0 con 0x7f2cd0074680 2023-07-14T18:23:20.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.433+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2cd002ca70 con 0x7f2cd0074680 2023-07-14T18:23:20.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.433+0000 7f2cc6ffd700 1 -- 172.21.15.64:0/4064204708 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2cc0099a10 con 0x7f2cd0074680 2023-07-14T18:23:20.485 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:23:20.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 msgr2=0x7f2cbc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 0x7f2cbc0a3680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098e0 tx=0x7f2ccc005fd0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 msgr2=0x7f2cd019c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2cc000e970 tx=0x7f2cc000ed30 comp rx=0 tx=0).stop 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 shutdown_connections 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2cd010c1e0 0x7f2cd01a3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2cbc0a11d0 0x7f2cbc0a3680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cd00730a0 0x7f2cd019bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 --2- 172.21.15.64:0/4064204708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cd0074680 0x7f2cd019c4a0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.488 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 >> 172.21.15.64:0/4064204708 conn(0x7f2cd00fb250 msgr2=0x7f2cd00fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.489 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 shutdown_connections 2023-07-14T18:23:20.489 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.435+0000 7f2cd81d8700 1 -- 172.21.15.64:0/4064204708 wait complete. 2023-07-14T18:23:20.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:23:20.628 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 -- 172.21.15.64:0/3033475046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27041006c0 msgr2=0x7f2704106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.628 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 --2- 172.21.15.64:0/3033475046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27041006c0 0x7f2704106f80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f26fc00b0a0 tx=0x7f26fc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 -- 172.21.15.64:0/3033475046 shutdown_connections 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 --2- 172.21.15.64:0/3033475046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27041006c0 0x7f2704106f80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 --2- 172.21.15.64:0/3033475046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27040feb40 0x7f27040fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 --2- 172.21.15.64:0/3033475046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 0x7f2704100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 -- 172.21.15.64:0/3033475046 >> 172.21.15.64:0/3033475046 conn(0x7f27040fa160 msgr2=0x7f27040fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 -- 172.21.15.64:0/3033475046 shutdown_connections 2023-07-14T18:23:20.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.576+0000 7f270afd5700 1 -- 172.21.15.64:0/3033475046 wait complete. 2023-07-14T18:23:20.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 Processor -- start 2023-07-14T18:23:20.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 -- start start 2023-07-14T18:23:20.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 0x7f2704195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27041006c0 0x7f270419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270419d490 con 0x7f27040feb40 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270419d5d0 con 0x7f27041006c0 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f270afd5700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270419d760 con 0x7f27040ffd30 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f2708d71700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f2708d71700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47168/0 (socket says 172.21.15.64:47168) 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f2708d71700 1 -- 172.21.15.64:0/2511145858 learned_addr learned my addr 172.21.15.64:0/2511145858 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.577+0000 7f2709572700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27041006c0 0x7f270419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2703fff700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 0x7f2704195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 -- 172.21.15.64:0/2511145858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 msgr2=0x7f2704195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 0x7f2704195e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 -- 172.21.15.64:0/2511145858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27041006c0 msgr2=0x7f270419cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27041006c0 0x7f270419cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 -- 172.21.15.64:0/2511145858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc009d00 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2708d71700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c990 tx=0x7f26f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8007b00 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270419da40 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f26f8007c60 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f8018660 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.578+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270419dfa0 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.579+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f270410a1d0 con 0x7f27040feb40 2023-07-14T18:23:20.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.580+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f26f800f450 con 0x7f27040feb40 2023-07-14T18:23:20.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.581+0000 7f2701ffb700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 0x7f26f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.581+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f26f80ccb40 con 0x7f27040feb40 2023-07-14T18:23:20.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.581+0000 7f2703fff700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 0x7f26f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.581+0000 7f2703fff700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 0x7f26f40a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009af0 tx=0x7f26f00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.585+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f26f80960a0 con 0x7f27040feb40 2023-07-14T18:23:20.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.784+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f270419e610 con 0x7f27040feb40 2023-07-14T18:23:20.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.784+0000 7f2701ffb700 1 -- 172.21.15.64:0/2511145858 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26f80996c0 con 0x7f27040feb40 2023-07-14T18:23:20.836 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:23:20.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 msgr2=0x7f26f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 0x7f26f40a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f26f0009af0 tx=0x7f26f00094f0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 msgr2=0x7f2704195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f26f800c990 tx=0x7f26f800cd50 comp rx=0 tx=0).stop 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 shutdown_connections 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f26f40a1150 0x7f26f40a3600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27041006c0 0x7f270419cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27040feb40 0x7f2704195910 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.786+0000 7f270afd5700 1 --2- 172.21.15.64:0/2511145858 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27040ffd30 0x7f2704195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.787+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 >> 172.21.15.64:0/2511145858 conn(0x7f27040fa160 msgr2=0x7f27040687d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.787+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 shutdown_connections 2023-07-14T18:23:20.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.787+0000 7f270afd5700 1 -- 172.21.15.64:0/2511145858 wait complete. 2023-07-14T18:23:20.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/34354472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:20.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3057958049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:20 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4064204708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:23:20.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:20 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/34354472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:20.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:20 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3057958049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:20 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4064204708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:20.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- 172.21.15.64:0/3784190979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 msgr2=0x7fa380107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 --2- 172.21.15.64:0/3784190979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa380107d30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa37800b0a0 tx=0x7fa37800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- 172.21.15.64:0/3784190979 shutdown_connections 2023-07-14T18:23:20.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 --2- 172.21.15.64:0/3784190979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa380107d30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 --2- 172.21.15.64:0/3784190979 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380100710 0x7fa380102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 --2- 172.21.15.64:0/3784190979 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380103030 0x7fa380105410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- 172.21.15.64:0/3784190979 >> 172.21.15.64:0/3784190979 conn(0x7fa3800fa130 msgr2=0x7fa3800fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:20.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- 172.21.15.64:0/3784190979 shutdown_connections 2023-07-14T18:23:20.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- 172.21.15.64:0/3784190979 wait complete. 2023-07-14T18:23:20.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 Processor -- start 2023-07-14T18:23:20.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.927+0000 7fa386390700 1 -- start start 2023-07-14T18:23:20.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380100710 0x7fa380100080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 0x7fa3800fe6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800ff3a0 con 0x7fa380105950 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3800ff510 con 0x7fa380103030 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa386390700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa38019f970 con 0x7fa380100710 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47186/0 (socket says 172.21.15.64:47186) 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 -- 172.21.15.64:0/3437591731 learned_addr learned my addr 172.21.15.64:0/3437591731 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:20.985 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa37ffff700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380100710 0x7fa380100080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa37f7fe700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 0x7fa3800fe6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 -- 172.21.15.64:0/3437591731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380100710 msgr2=0x7fa380100080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380100710 0x7fa380100080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 -- 172.21.15.64:0/3437591731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 msgr2=0x7fa3800fe6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 0x7fa3800fe6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 -- 172.21.15.64:0/3437591731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa378009d00 con 0x7fa380105950 2023-07-14T18:23:20.986 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa37f7fe700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 0x7fa3800fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.928+0000 7fa38492d700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa3780062a0 tx=0x7fa378004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.929+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37800e040 con 0x7fa380105950 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.929+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa378003e90 con 0x7fa380105950 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.929+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa38019fbf0 con 0x7fa380105950 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.929+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa378004020 con 0x7fa380105950 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.929+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3801a0020 con 0x7fa380105950 2023-07-14T18:23:20.987 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.930+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3800fbca0 con 0x7fa380105950 2023-07-14T18:23:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.931+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa378003a00 con 0x7fa380105950 2023-07-14T18:23:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.931+0000 7fa37d7fa700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 0x7fa3680a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.932+0000 7fa37ffff700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 0x7fa3680a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.932+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa3780ce790 con 0x7fa380105950 2023-07-14T18:23:20.988 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.932+0000 7fa37ffff700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 0x7fa3680a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa370005950 tx=0x7fa37000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:20.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:20.936+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa378097d80 con 0x7fa380105950 2023-07-14T18:23:21.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:20 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/34354472' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:21.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:20 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3057958049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:20 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4064204708' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.136+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa3800612f0 con 0x7fa380105950 2023-07-14T18:23:21.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.136+0000 7fa37d7fa700 1 -- 172.21.15.64:0/3437591731 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa37809b3a0 con 0x7fa380105950 2023-07-14T18:23:21.188 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:23:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 msgr2=0x7fa3680a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 0x7fa3680a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa370005950 tx=0x7fa37000b410 comp rx=0 tx=0).stop 2023-07-14T18:23:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 msgr2=0x7fa3800fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa3780062a0 tx=0x7fa378004d00 comp rx=0 tx=0).stop 2023-07-14T18:23:21.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 shutdown_connections 2023-07-14T18:23:21.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa380100710 0x7fa380100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa3680a1150 0x7fa3680a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa380105950 0x7fa3800fec00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 --2- 172.21.15.64:0/3437591731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa380103030 0x7fa3800fe6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 >> 172.21.15.64:0/3437591731 conn(0x7fa3800fa130 msgr2=0x7fa380101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:21.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 shutdown_connections 2023-07-14T18:23:21.193 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.138+0000 7fa386390700 1 -- 172.21.15.64:0/3437591731 wait complete. 2023-07-14T18:23:21.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:23:21.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/3024739009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 msgr2=0x7f0f74100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/3024739009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f74100150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0f64009990 tx=0x7f0f64009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/3024739009 shutdown_connections 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/3024739009 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 0x7f0f740fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/3024739009 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 0x7f0f74106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/3024739009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f74100150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/3024739009 >> 172.21.15.64:0/3024739009 conn(0x7f0f740fa130 msgr2=0x7f0f740fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:21.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/3024739009 shutdown_connections 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.277+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/3024739009 wait complete. 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 Processor -- start 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 -- start start 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 0x7f0f74068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 0x7f0f7419cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7419d440 con 0x7f0f740ffd00 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7419d5b0 con 0x7f0f74100690 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7c7c4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7419d770 con 0x7f0f740feb10 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47198/0 (socket says 172.21.15.64:47198) 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 -- 172.21.15.64:0/776627485 learned_addr learned my addr 172.21.15.64:0/776627485 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7a560700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 0x7f0f74068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 -- 172.21.15.64:0/776627485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 msgr2=0x7f0f74068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 0x7f0f74068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f79d5f700 1 -- 172.21.15.64:0/776627485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 msgr2=0x7f0f7419cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.278+0000 7f0f7ad61700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 0x7f0f7419cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f79d5f700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 0x7f0f7419cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f79d5f700 1 -- 172.21.15.64:0/776627485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f640095f0 con 0x7f0f740ffd00 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f79d5f700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0f640098e0 tx=0x7f0f64005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6401d070 con 0x7f0f740ffd00 2023-07-14T18:23:21.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f6400bed0 con 0x7f0f740ffd00 2023-07-14T18:23:21.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7419d9f0 con 0x7f0f740ffd00 2023-07-14T18:23:21.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6400f460 con 0x7f0f740ffd00 2023-07-14T18:23:21.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.279+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7419de70 con 0x7f0f740ffd00 2023-07-14T18:23:21.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.280+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f741061a0 con 0x7f0f740ffd00 2023-07-14T18:23:21.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.281+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f6400ba40 con 0x7f0f740ffd00 2023-07-14T18:23:21.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.282+0000 7f0f6b7fe700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 0x7f0f600a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.282+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0f640ce170 con 0x7f0f740ffd00 2023-07-14T18:23:21.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.282+0000 7f0f7a560700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 0x7f0f600a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.282+0000 7f0f7a560700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 0x7f0f600a3680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005950 tx=0x7f0f700058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:21.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.286+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f64097700 con 0x7f0f740ffd00 2023-07-14T18:23:21.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.502+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0f7402c8e0 con 0x7f0f740ffd00 2023-07-14T18:23:21.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.503+0000 7f0f6b7fe700 1 -- 172.21.15.64:0/776627485 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+589 (secure 0 0 0) 0x7f0f6409ad20 con 0x7f0f740ffd00 2023-07-14T18:23:21.555 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:21.555 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:21.555 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2932389061 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:21.555 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3783129146 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1024618281 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/377975996 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1532744554 2023-07-15T18:16:37.487452+0000 2023-07-14T18:23:21.556 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2890765683 2023-07-15T18:17:05.380198+0000 2023-07-14T18:23:21.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 msgr2=0x7f0f600a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 0x7f0f600a3680 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0f70005950 tx=0x7f0f700058e0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 msgr2=0x7f0f7419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0f640098e0 tx=0x7f0f64005ef0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 shutdown_connections 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f740feb10 0x7f0f74068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f600a11d0 0x7f0f600a3680 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f74100690 0x7f0f7419cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 --2- 172.21.15.64:0/776627485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f740ffd00 0x7f0f7419a660 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 >> 172.21.15.64:0/776627485 conn(0x7f0f740fa130 msgr2=0x7f0f741032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:21.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 shutdown_connections 2023-07-14T18:23:21.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.505+0000 7f0f7c7c4700 1 -- 172.21.15.64:0/776627485 wait complete. 2023-07-14T18:23:21.559 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2023-07-14T18:23:21.569 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi064.front.sepia.ceph.com... 2023-07-14T18:23:21.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:21.570 DEBUG:teuthology.orchestra.run.smithi064:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-14T18:23:21.606 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi064.stderr:ceph-fuse[134022]: fuse finished with error 0 and tester_r 0 2023-07-14T18:23:21.607 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T18:23:21.685 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:23:21.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:23:21.687 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-14T18:23:21.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:21.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:21 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2511145858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:21 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3437591731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:21 smithi064 ceph-mon[96720]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 278 B/s wr, 0 op/s 2023-07-14T18:23:21.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:21 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/776627485' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:21.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 -- 172.21.15.64:0/475271431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64500feb00 msgr2=0x7f64500fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 --2- 172.21.15.64:0/475271431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64500feb00 0x7f64500fef10 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f6440009990 tx=0x7f6440009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 -- 172.21.15.64:0/475271431 shutdown_connections 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 --2- 172.21.15.64:0/475271431 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6450100680 0x7f6450106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 --2- 172.21.15.64:0/475271431 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64500feb00 0x7f64500fef10 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 --2- 172.21.15.64:0/475271431 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 -- 172.21.15.64:0/475271431 >> 172.21.15.64:0/475271431 conn(0x7f64500fa100 msgr2=0x7f64500fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 -- 172.21.15.64:0/475271431 shutdown_connections 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.826+0000 7f64565db700 1 -- 172.21.15.64:0/475271431 wait complete. 2023-07-14T18:23:21.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 Processor -- start 2023-07-14T18:23:21.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 -- start start 2023-07-14T18:23:21.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 0x7f6450197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645019f6a0 con 0x7f6450100680 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645019f7e0 con 0x7f64500ffcf0 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f64565db700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645019f970 con 0x7f64500feb00 2023-07-14T18:23:21.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47214/0 (socket says 172.21.15.64:47214) 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 -- 172.21.15.64:0/2565805486 learned_addr learned my addr 172.21.15.64:0/2565805486 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f644f7fe700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f644ffff700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 0x7f6450197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 -- 172.21.15.64:0/2565805486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 msgr2=0x7f6450197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 0x7f6450197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 -- 172.21.15.64:0/2565805486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 msgr2=0x7f6450198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f6454b78700 1 -- 172.21.15.64:0/2565805486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64400095f0 con 0x7f6450100680 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f644ffff700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 0x7f6450197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.827+0000 7f644f7fe700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f6454b78700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6448007ee0 tx=0x7f644800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448017070 con 0x7f6450100680 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f644800f040 con 0x7f6450100680 2023-07-14T18:23:21.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645019fc50 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6448005710 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.828+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64501a0110 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.829+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645010a1d0 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.830+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6448005870 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.831+0000 7f644d7fa700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 0x7f64380a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.831+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f64480cc960 con 0x7f6450100680 2023-07-14T18:23:21.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.831+0000 7f644ffff700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 0x7f64380a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:21.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.831+0000 7f644ffff700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 0x7f64380a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6440006010 tx=0x7f6440009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:21.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:21.834+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6448095ec0 con 0x7f6450100680 2023-07-14T18:23:21.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:21 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2511145858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:21 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3437591731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:21.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:21 smithi144 ceph-mon[98417]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 278 B/s wr, 0 op/s 2023-07-14T18:23:21.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:21 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/776627485' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:22.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:21 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2511145858' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:23:22.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:21 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3437591731' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:23:22.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:21 smithi105 ceph-mon[98013]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 278 B/s wr, 0 op/s 2023-07-14T18:23:22.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:21 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/776627485' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:23:22.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.036+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64501a06c0 con 0x7f6450100680 2023-07-14T18:23:22.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.037+0000 7f644d7fa700 1 -- 172.21.15.64:0/2565805486 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f64480994e0 con 0x7f6450100680 2023-07-14T18:23:22.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:22.090 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 msgr2=0x7f64380a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 0x7f64380a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6440006010 tx=0x7f6440009530 comp rx=0 tx=0).stop 2023-07-14T18:23:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 msgr2=0x7f645019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6448007ee0 tx=0x7f644800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 shutdown_connections 2023-07-14T18:23:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64500feb00 0x7f6450197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64380a11a0 0x7f64380a3650 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6450100680 0x7f645019f0d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 --2- 172.21.15.64:0/2565805486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64500ffcf0 0x7f6450198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 >> 172.21.15.64:0/2565805486 conn(0x7f64500fa100 msgr2=0x7f6450067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:22.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 shutdown_connections 2023-07-14T18:23:22.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.040+0000 7f64565db700 1 -- 172.21.15.64:0/2565805486 wait complete. 2023-07-14T18:23:22.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:22.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.177+0000 7fa10bebc700 1 -- 172.21.15.64:0/226899470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fe470 msgr2=0x7fa1040fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.177+0000 7fa10bebc700 1 --2- 172.21.15.64:0/226899470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fe470 0x7fa1040fe880 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa100009990 tx=0x7fa100009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- 172.21.15.64:0/226899470 shutdown_connections 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- 172.21.15.64:0/226899470 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040ff770 0x7fa1040ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- 172.21.15.64:0/226899470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fe470 0x7fa1040fe880 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- 172.21.15.64:0/226899470 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040fedc0 0x7fa1040ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- 172.21.15.64:0/226899470 >> 172.21.15.64:0/226899470 conn(0x7fa1040fa100 msgr2=0x7fa1040fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:22.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- 172.21.15.64:0/226899470 shutdown_connections 2023-07-14T18:23:22.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- 172.21.15.64:0/226899470 wait complete. 2023-07-14T18:23:22.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 Processor -- start 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- start start 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040fe470 0x7fa104197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 0x7fa10419f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10419f600 con 0x7fa1040fedc0 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10419f740 con 0x7fa1040fe470 2023-07-14T18:23:22.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.178+0000 7fa10bebc700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10419f8d0 con 0x7fa1040ff770 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109c58700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040fe470 0x7fa104197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47226/0 (socket says 172.21.15.64:47226) 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 -- 172.21.15.64:0/3487559907 learned_addr learned my addr 172.21.15.64:0/3487559907 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa10a459700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 0x7fa10419f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 -- 172.21.15.64:0/3487559907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 msgr2=0x7fa10419f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 0x7fa10419f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 -- 172.21.15.64:0/3487559907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040fe470 msgr2=0x7fa104197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040fe470 0x7fa104197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 -- 172.21.15.64:0/3487559907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000095f0 con 0x7fa1040fedc0 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa10a459700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 0x7fa10419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa109457700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400b530 tx=0x7fa0f400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4011830 con 0x7fa1040fedc0 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa0f4011e20 con 0x7fa1040fedc0 2023-07-14T18:23:22.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10419fbb0 con 0x7fa1040fedc0 2023-07-14T18:23:22.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400f2e0 con 0x7fa1040fedc0 2023-07-14T18:23:22.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.179+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1041a0110 con 0x7fa1040fedc0 2023-07-14T18:23:22.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.180+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104109250 con 0x7fa1040fedc0 2023-07-14T18:23:22.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.181+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa0f4011990 con 0x7fa1040fedc0 2023-07-14T18:23:22.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.182+0000 7fa0faffd700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 0x7fa0f00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.182+0000 7fa109c58700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 0x7fa0f00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.182+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa0f40cc5f0 con 0x7fa1040fedc0 2023-07-14T18:23:22.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.182+0000 7fa109c58700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 0x7fa0f00a36d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b5c0 tx=0x7fa100005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:22.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.186+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa0f4095af0 con 0x7fa1040fedc0 2023-07-14T18:23:22.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.389+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa10402c8e0 con 0x7fa1040fedc0 2023-07-14T18:23:22.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.390+0000 7fa0faffd700 1 -- 172.21.15.64:0/3487559907 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7fa0f4099110 con 0x7fa1040fedc0 2023-07-14T18:23:22.442 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:22.443 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:22.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 msgr2=0x7fa0f00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 0x7fa0f00a36d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b5c0 tx=0x7fa100005fb0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 msgr2=0x7fa104197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400b530 tx=0x7fa0f400b840 comp rx=0 tx=0).stop 2023-07-14T18:23:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 shutdown_connections 2023-07-14T18:23:22.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa1040ff770 0x7fa10419f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa0f00a1220 0x7fa0f00a36d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa1040fe470 0x7fa104197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 --2- 172.21.15.64:0/3487559907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1040fedc0 0x7fa104197fc0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 >> 172.21.15.64:0/3487559907 conn(0x7fa1040fa100 msgr2=0x7fa1040fbed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:22.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 shutdown_connections 2023-07-14T18:23:22.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.393+0000 7fa10bebc700 1 -- 172.21.15.64:0/3487559907 wait complete. 2023-07-14T18:23:22.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T18:23:22.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 -- 172.21.15.64:0/2515662854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 msgr2=0x7f0f48106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/2515662854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48106f50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000b0a0 tx=0x7f0f4000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 -- 172.21.15.64:0/2515662854 shutdown_connections 2023-07-14T18:23:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/2515662854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f48106f50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/2515662854 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 0x7f0f480fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/2515662854 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 -- 172.21.15.64:0/2515662854 >> 172.21.15.64:0/2515662854 conn(0x7f0f480fa130 msgr2=0x7f0f480fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 -- 172.21.15.64:0/2515662854 shutdown_connections 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.535+0000 7f0f4eba5700 1 -- 172.21.15.64:0/2515662854 wait complete. 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 Processor -- start 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 -- start start 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 0x7f0f48194910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48194e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819c490 con 0x7f0f48100690 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819c5d0 con 0x7f0f480feb10 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4eba5700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f4819c760 con 0x7f0f480ffd00 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47240/0 (socket says 172.21.15.64:47240) 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 -- 172.21.15.64:0/1615492516 learned_addr learned my addr 172.21.15.64:0/1615492516 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4c941700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 0x7f0f48194910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f47fff700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48194e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 -- 172.21.15.64:0/1615492516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 msgr2=0x7f0f48194e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48194e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 -- 172.21.15.64:0/1615492516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 msgr2=0x7f0f48194910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 0x7f0f48194910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 -- 172.21.15.64:0/1615492516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f40009d00 con 0x7f0f48100690 2023-07-14T18:23:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f47fff700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.536+0000 7f0f4d142700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009fd0 tx=0x7f0f4000ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.537+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4000e040 con 0x7f0f48100690 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.537+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0f40003c00 con 0x7f0f48100690 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.537+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f480687d0 con 0x7f0f48100690 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.537+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f400052a0 con 0x7f0f48100690 2023-07-14T18:23:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.537+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48068d30 con 0x7f0f48100690 2023-07-14T18:23:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.538+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f480696c0 con 0x7f0f48100690 2023-07-14T18:23:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.539+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0f4001a4c0 con 0x7f0f48100690 2023-07-14T18:23:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.539+0000 7f0f45ffb700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.540+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0f40020030 con 0x7f0f48100690 2023-07-14T18:23:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.540+0000 7f0f4c941700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.540+0000 7f0f4c941700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f480ffb60 tx=0x7f0f3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.543+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0f4005d780 con 0x7f0f48100690 2023-07-14T18:23:22.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:22.787+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f0f48105f40 con 0x7f0f48100690 2023-07-14T18:23:22.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:22 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2565805486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:22.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:22 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3487559907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:22.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:22 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2565805486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:22.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:22 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3487559907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:23.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:22 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2565805486' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:23.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:22 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3487559907' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:23.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.537+0000 7f0f45ffb700 1 -- 172.21.15.64:0/1615492516 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0f400991a0 con 0x7f0f48100690 2023-07-14T18:23:23.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 msgr2=0x7f0f380a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0f480ffb60 tx=0x7f0f3c009260 comp rx=0 tx=0).stop 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 msgr2=0x7f0f4819bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0f40009fd0 tx=0x7f0f4000ba50 comp rx=0 tx=0).stop 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 shutdown_connections 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0f380a11d0 0x7f0f380a3680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f48100690 0x7f0f4819bec0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f480feb10 0x7f0f48194910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 --2- 172.21.15.64:0/1615492516 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0f480ffd00 0x7f0f48194e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.540+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 >> 172.21.15.64:0/1615492516 conn(0x7f0f480fa130 msgr2=0x7f0f481031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.541+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 shutdown_connections 2023-07-14T18:23:23.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.541+0000 7f0f4eba5700 1 -- 172.21.15.64:0/1615492516 wait complete. 2023-07-14T18:23:23.607 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T18:23:23.607 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2023-07-14T18:23:23.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 -- 172.21.15.64:0/2875765010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4750100690 msgr2=0x7f4750106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 --2- 172.21.15.64:0/2875765010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4750100690 0x7f4750106f50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f474800b0a0 tx=0x7f474800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 -- 172.21.15.64:0/2875765010 shutdown_connections 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 --2- 172.21.15.64:0/2875765010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4750100690 0x7f4750106f50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 --2- 172.21.15.64:0/2875765010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47500feb10 0x7f47500fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 --2- 172.21.15.64:0/2875765010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500ffd00 0x7f4750100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.691+0000 7f4756fbc700 1 -- 172.21.15.64:0/2875765010 >> 172.21.15.64:0/2875765010 conn(0x7f47500fa130 msgr2=0x7f47500fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- 172.21.15.64:0/2875765010 shutdown_connections 2023-07-14T18:23:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- 172.21.15.64:0/2875765010 wait complete. 2023-07-14T18:23:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 Processor -- start 2023-07-14T18:23:23.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- start start 2023-07-14T18:23:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500feb10 0x7f4750068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 0x7f475019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475019d4d0 con 0x7f47500ffd00 2023-07-14T18:23:23.747 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475019d640 con 0x7f4750100690 2023-07-14T18:23:23.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f4756fbc700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475019d800 con 0x7f47500feb10 2023-07-14T18:23:23.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f474ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:23.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.692+0000 7f474ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47252/0 (socket says 172.21.15.64:47252) 2023-07-14T18:23:23.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 -- 172.21.15.64:0/868917031 learned_addr learned my addr 172.21.15.64:0/868917031 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:23.748 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f4754d58700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500feb10 0x7f4750068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:23.749 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f4755559700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 0x7f475019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:23.749 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 -- 172.21.15.64:0/868917031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500feb10 msgr2=0x7f4750068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:23.749 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500feb10 0x7f4750068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.749 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 -- 172.21.15.64:0/868917031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 msgr2=0x7f475019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:23.750 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 0x7f475019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:23.750 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 -- 172.21.15.64:0/868917031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748009d00 con 0x7f47500ffd00 2023-07-14T18:23:23.750 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f4755559700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 0x7f475019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:23.750 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474ffff700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f473c00d950 tx=0x7f473c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c009940 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f473c010460 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475019dae0 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c00f620 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.693+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475019dff0 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.695+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4750105a40 con 0x7f47500ffd00 2023-07-14T18:23:23.751 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.695+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f473c0105d0 con 0x7f47500ffd00 2023-07-14T18:23:23.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.696+0000 7f474dffb700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 0x7f47400a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:23.752 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.696+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f473c0cc950 con 0x7f47500ffd00 2023-07-14T18:23:23.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.696+0000 7f4754d58700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 0x7f47400a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:23.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.697+0000 7f4754d58700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 0x7f47400a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47500ffb60 tx=0x7f4744009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:23.753 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.700+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f473c095eb0 con 0x7f47500ffd00 2023-07-14T18:23:23.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:23 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:23:23.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:23 smithi064 ceph-mon[96720]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:23:23.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:23 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:23:23.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:23 smithi144 ceph-mon[98417]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:23:23.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:23.942+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f475002ccb0 con 0x7f47500ffd00 2023-07-14T18:23:24.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:23 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T18:23:24.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:23 smithi105 ceph-mon[98013]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-07-14T18:23:24.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.232+0000 7f474dffb700 1 -- 172.21.15.64:0/868917031 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f473c0160b0 con 0x7f47500ffd00 2023-07-14T18:23:24.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 msgr2=0x7f47400a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:24.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 0x7f47400a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f47500ffb60 tx=0x7f4744009260 comp rx=0 tx=0).stop 2023-07-14T18:23:24.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 msgr2=0x7f475019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:24.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f473c00d950 tx=0x7f473c00dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:24.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 shutdown_connections 2023-07-14T18:23:24.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47500feb10 0x7f4750068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47400a1150 0x7f47400a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.289 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4750100690 0x7f475019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 --2- 172.21.15.64:0/868917031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47500ffd00 0x7f475019a6f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 >> 172.21.15.64:0/868917031 conn(0x7f47500fa130 msgr2=0x7f47500fc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:24.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 shutdown_connections 2023-07-14T18:23:24.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.235+0000 7f4756fbc700 1 -- 172.21.15.64:0/868917031 wait complete. 2023-07-14T18:23:24.304 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T18:23:24.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.385+0000 7f201956f700 1 -- 172.21.15.64:0/3704014567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20140ffd60 msgr2=0x7f20141001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.385+0000 7f201956f700 1 --2- 172.21.15.64:0/3704014567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20140ffd60 0x7f20141001b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2008009990 tx=0x7f2008009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 -- 172.21.15.64:0/3704014567 shutdown_connections 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 --2- 172.21.15.64:0/3704014567 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20141006f0 0x7f2014106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 --2- 172.21.15.64:0/3704014567 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 0x7f20140fef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 --2- 172.21.15.64:0/3704014567 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20140ffd60 0x7f20141001b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 -- 172.21.15.64:0/3704014567 >> 172.21.15.64:0/3704014567 conn(0x7f20140fa1d0 msgr2=0x7f20140fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:24.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 -- 172.21.15.64:0/3704014567 shutdown_connections 2023-07-14T18:23:24.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 -- 172.21.15.64:0/3704014567 wait complete. 2023-07-14T18:23:24.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 Processor -- start 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.386+0000 7f201956f700 1 -- start start 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 0x7f20141947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20140ffd60 0x7f2014194cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201419c330 con 0x7f20141006f0 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201419c470 con 0x7f20140feb70 2023-07-14T18:23:24.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201419c600 con 0x7f20140ffd60 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47280/0 (socket says 172.21.15.64:47280) 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 -- 172.21.15.64:0/3869595557 learned_addr learned my addr 172.21.15.64:0/3869595557 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f2012ffd700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 0x7f20141947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20127fc700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20140ffd60 0x7f2014194cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 -- 172.21.15.64:0/3869595557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20140ffd60 msgr2=0x7f2014194cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20140ffd60 0x7f2014194cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 -- 172.21.15.64:0/3869595557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 msgr2=0x7f20141947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 0x7f20141947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 -- 172.21.15.64:0/3869595557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080095f0 con 0x7f20141006f0 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f20137fe700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f200c007ee0 tx=0x7f200c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c017070 con 0x7f20141006f0 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f200c00f040 con 0x7f20141006f0 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f201419c8e0 con 0x7f20141006f0 2023-07-14T18:23:24.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.387+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c005570 con 0x7f20141006f0 2023-07-14T18:23:24.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.388+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f201419cdf0 con 0x7f20141006f0 2023-07-14T18:23:24.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.388+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2014105b20 con 0x7f20141006f0 2023-07-14T18:23:24.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.389+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f200c00db20 con 0x7f20141006f0 2023-07-14T18:23:24.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.390+0000 7f1ffbfff700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 0x7f1ffc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:24.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.390+0000 7f2012ffd700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 0x7f1ffc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:24.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.390+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f200c0cc190 con 0x7f20141006f0 2023-07-14T18:23:24.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.390+0000 7f2012ffd700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 0x7f1ffc0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f20140ffbc0 tx=0x7f200400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:24.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.394+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f200c095790 con 0x7f20141006f0 2023-07-14T18:23:24.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:24.642+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f201402c8e0 con 0x7f20141006f0 2023-07-14T18:23:24.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:23:24.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:24 smithi064 ceph-mon[96720]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:23:24.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:23:24.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:24 smithi144 ceph-mon[98417]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:23:25.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1615492516' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T18:23:25.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T18:23:25.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2023-07-14T18:23:25.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T18:23:25.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T18:23:25.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/868917031' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2023-07-14T18:23:25.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T18:23:25.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:24 smithi105 ceph-mon[98013]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T18:23:25.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.238+0000 7f1ffbfff700 1 -- 172.21.15.64:0/3869595557 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f200c098db0 con 0x7f20141006f0 2023-07-14T18:23:25.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 msgr2=0x7f1ffc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:25.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 0x7f1ffc0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f20140ffbc0 tx=0x7f200400b410 comp rx=0 tx=0).stop 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 msgr2=0x7f201419bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f200c007ee0 tx=0x7f200c00d3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 shutdown_connections 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1ffc0a1150 0x7f1ffc0a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20141006f0 0x7f201419bd60 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20140feb70 0x7f20141947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 --2- 172.21.15.64:0/3869595557 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20140ffd60 0x7f2014194cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.242+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 >> 172.21.15.64:0/3869595557 conn(0x7f20140fa1d0 msgr2=0x7f20140fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:25.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.243+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 shutdown_connections 2023-07-14T18:23:25.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.243+0000 7f201956f700 1 -- 172.21.15.64:0/3869595557 wait complete. 2023-07-14T18:23:25.314 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 -- 172.21.15.64:0/2403456743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a44100690 msgr2=0x7f5a44106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 --2- 172.21.15.64:0/2403456743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a44100690 0x7f5a44106f50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c00b0a0 tx=0x7f5a3c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 -- 172.21.15.64:0/2403456743 shutdown_connections 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 --2- 172.21.15.64:0/2403456743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a44100690 0x7f5a44106f50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 --2- 172.21.15.64:0/2403456743 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440feb10 0x7f5a440fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 --2- 172.21.15.64:0/2403456743 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a440ffd00 0x7f5a44100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 -- 172.21.15.64:0/2403456743 >> 172.21.15.64:0/2403456743 conn(0x7f5a440fa130 msgr2=0x7f5a440fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 -- 172.21.15.64:0/2403456743 shutdown_connections 2023-07-14T18:23:25.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 -- 172.21.15.64:0/2403456743 wait complete. 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.401+0000 7f5a489c0700 1 Processor -- start 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 -- start start 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 0x7f5a44195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a44100690 0x7f5a4419cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:25.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4419d520 con 0x7f5a440feb10 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4419d660 con 0x7f5a440ffd00 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a489c0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4419d7f0 con 0x7f5a44100690 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47292/0 (socket says 172.21.15.64:47292) 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 -- 172.21.15.64:0/1308801672 learned_addr learned my addr 172.21.15.64:0/1308801672 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a41d9b700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 0x7f5a44195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:25.457 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a42d9d700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a44100690 0x7f5a4419cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 -- 172.21.15.64:0/1308801672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a44100690 msgr2=0x7f5a4419cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a44100690 0x7f5a4419cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 -- 172.21.15.64:0/1308801672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 msgr2=0x7f5a44195ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 0x7f5a44195ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 -- 172.21.15.64:0/1308801672 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a3c009d00 con 0x7f5a440feb10 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a41d9b700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 0x7f5a44195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a4259c700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400d980 tx=0x7f5a3400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:25.458 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.402+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400bb60 con 0x7f5a440feb10 2023-07-14T18:23:25.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.403+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5a34007620 con 0x7f5a440feb10 2023-07-14T18:23:25.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.403+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4419dad0 con 0x7f5a440feb10 2023-07-14T18:23:25.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.403+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a3400f3c0 con 0x7f5a440feb10 2023-07-14T18:23:25.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.403+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4419e030 con 0x7f5a440feb10 2023-07-14T18:23:25.459 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.403+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4410a1a0 con 0x7f5a440feb10 2023-07-14T18:23:25.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.405+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5a34010460 con 0x7f5a440feb10 2023-07-14T18:23:25.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.405+0000 7f5a337fe700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 0x7f5a2c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:25.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.406+0000 7f5a41d9b700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 0x7f5a2c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:25.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.406+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5a340cd2c0 con 0x7f5a440feb10 2023-07-14T18:23:25.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.406+0000 7f5a41d9b700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 0x7f5a2c0a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5a38005e10 tx=0x7f5a38005d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:25.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.409+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5a34096820 con 0x7f5a440feb10 2023-07-14T18:23:25.590 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:23:25.278+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T18:23:25.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:25.659+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f5a4410a390 con 0x7f5a440feb10 2023-07-14T18:23:25.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:active 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T18:23:25.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T18:23:25.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:25 smithi064 ceph-mon[96720]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 0 op/s 2023-07-14T18:23:25.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:23:25.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:active 2023-07-14T18:23:25.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:23:25.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:23:25.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T18:23:25.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T18:23:25.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T18:23:25.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:25 smithi144 ceph-mon[98417]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 0 op/s 2023-07-14T18:23:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T18:23:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] up:active 2023-07-14T18:23:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3869595557' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T18:23:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T18:23:26.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T18:23:26.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T18:23:26.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T18:23:26.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:25 smithi105 ceph-mon[98013]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 0 op/s 2023-07-14T18:23:26.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.248+0000 7f5a337fe700 1 -- 172.21.15.64:0/1308801672 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5a34099e40 con 0x7f5a440feb10 2023-07-14T18:23:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 msgr2=0x7f5a2c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:26.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 0x7f5a2c0a3680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5a38005e10 tx=0x7f5a38005d60 comp rx=0 tx=0).stop 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 msgr2=0x7f5a441959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5a3400d980 tx=0x7f5a3400dd40 comp rx=0 tx=0).stop 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 shutdown_connections 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5a44100690 0x7f5a4419cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5a2c0a11d0 0x7f5a2c0a3680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a440feb10 0x7f5a441959a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 --2- 172.21.15.64:0/1308801672 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5a440ffd00 0x7f5a44195ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 >> 172.21.15.64:0/1308801672 conn(0x7f5a440fa130 msgr2=0x7f5a44067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.251+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 shutdown_connections 2023-07-14T18:23:26.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.252+0000 7f5a489c0700 1 -- 172.21.15.64:0/1308801672 wait complete. 2023-07-14T18:23:26.322 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-14T18:23:26.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T18:23:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- 172.21.15.64:0/3797575559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 msgr2=0x7f1804100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:26.460 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3797575559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f1804100590 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f17f8009990 tx=0x7f17f8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- 172.21.15.64:0/3797575559 shutdown_connections 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3797575559 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 0x7f18040feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3797575559 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f1804100590 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3797575559 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 0x7f18040fe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- 172.21.15.64:0/3797575559 >> 172.21.15.64:0/3797575559 conn(0x7f18040f98a0 msgr2=0x7f18040fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:26.461 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- 172.21.15.64:0/3797575559 shutdown_connections 2023-07-14T18:23:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- 172.21.15.64:0/3797575559 wait complete. 2023-07-14T18:23:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 Processor -- start 2023-07-14T18:23:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- start start 2023-07-14T18:23:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 0x7f1804197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:26.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 0x7f1804197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180419f5d0 con 0x7f1804100180 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180419f710 con 0x7f18040fdc70 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.407+0000 7f180b1d4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180419f8a0 con 0x7f18040fe620 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47310/0 (socket says 172.21.15.64:47310) 2023-07-14T18:23:26.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 -- 172.21.15.64:0/3283376148 learned_addr learned my addr 172.21.15.64:0/3283376148 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1803fff700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 0x7f1804197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1808f70700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 0x7f1804197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 -- 172.21.15.64:0/3283376148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 msgr2=0x7f1804197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 0x7f1804197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 -- 172.21.15.64:0/3283376148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 msgr2=0x7f1804197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 0x7f1804197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:26.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 -- 172.21.15.64:0/3283376148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f80095f0 con 0x7f1804100180 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1809771700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17fc007ee0 tx=0x7f17fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc017070 con 0x7f1804100180 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f17fc00f040 con 0x7f1804100180 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180419fb80 con 0x7f1804100180 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.408+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17fc005570 con 0x7f1804100180 2023-07-14T18:23:26.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.409+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18041a0040 con 0x7f1804100180 2023-07-14T18:23:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.409+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1804109b20 con 0x7f1804100180 2023-07-14T18:23:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.410+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17fc00db20 con 0x7f1804100180 2023-07-14T18:23:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.411+0000 7f1801ffb700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 0x7f17f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:26.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.411+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f17fc0cc1b0 con 0x7f1804100180 2023-07-14T18:23:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.411+0000 7f1808f70700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 0x7f17f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.411+0000 7f1808f70700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 0x7f17f40a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17f800b5c0 tx=0x7f17f8009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:26.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.415+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f17fc0957b0 con 0x7f1804100180 2023-07-14T18:23:26.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:26.651+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f1804060b10 con 0x7f17f40a1150 2023-07-14T18:23:26.841 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:23:26.657+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:23:26.842 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:23:26.657+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:23:26.842 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:23:26.657+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:23:26.842 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:23:26.657+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:23:26.842 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:23:26.657+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:active 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T18:23:26.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T18:23:26.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:26 smithi064 ceph-mon[96720]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T18:23:26.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:23:26.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:active 2023-07-14T18:23:26.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:23:26.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T18:23:26.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T18:23:26.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T18:23:26.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:26 smithi144 ceph-mon[98417]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T18:23:27.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T18:23:27.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] up:active 2023-07-14T18:23:27.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1308801672' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T18:23:27.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2023-07-14T18:23:27.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2023-07-14T18:23:27.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:creating} 8 up:standby 2023-07-14T18:23:27.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:26 smithi105 ceph-mon[98013]: daemon mds.c is now active in filesystem cephfs as rank 3 2023-07-14T18:23:27.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.296+0000 7f1801ffb700 1 -- 172.21.15.64:0/3283376148 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1804060b10 con 0x7f17f40a1150 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 msgr2=0x7f17f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 0x7f17f40a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f17f800b5c0 tx=0x7f17f8009f00 comp rx=0 tx=0).stop 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 msgr2=0x7f180419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f17fc007ee0 tx=0x7f17fc00d3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 shutdown_connections 2023-07-14T18:23:27.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17f40a1150 0x7f17f40a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1804100180 0x7f180419f000 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18040fdc70 0x7f1804197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 --2- 172.21.15.64:0/3283376148 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18040fe620 0x7f1804197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 >> 172.21.15.64:0/3283376148 conn(0x7f18040f98a0 msgr2=0x7f18040fb690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 shutdown_connections 2023-07-14T18:23:27.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.299+0000 7f180b1d4700 1 -- 172.21.15.64:0/3283376148 wait complete. 2023-07-14T18:23:27.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T18:23:27.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 -- 172.21.15.64:0/4077683306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600feb30 msgr2=0x7fd6600fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 --2- 172.21.15.64:0/4077683306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600feb30 0x7fd6600fef40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd650009990 tx=0x7fd650009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 -- 172.21.15.64:0/4077683306 shutdown_connections 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 --2- 172.21.15.64:0/4077683306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6601006b0 0x7fd660106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 --2- 172.21.15.64:0/4077683306 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600feb30 0x7fd6600fef40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 --2- 172.21.15.64:0/4077683306 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6600ffd20 0x7fd660100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 -- 172.21.15.64:0/4077683306 >> 172.21.15.64:0/4077683306 conn(0x7fd6600fa190 msgr2=0x7fd6600fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:27.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 -- 172.21.15.64:0/4077683306 shutdown_connections 2023-07-14T18:23:27.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.446+0000 7fd664d3c700 1 -- 172.21.15.64:0/4077683306 wait complete. 2023-07-14T18:23:27.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 Processor -- start 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 -- start start 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600ffd20 0x7fd660101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6601006b0 0x7fd660101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660101de0 con 0x7fd6600feb30 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660101f50 con 0x7fd6600ffd20 2023-07-14T18:23:27.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd664d3c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66019f940 con 0x7fd6601006b0 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd65e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47340/0 (socket says 172.21.15.64:47340) 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd65e59c700 1 -- 172.21.15.64:0/2040541144 learned_addr learned my addr 172.21.15.64:0/2040541144 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.447+0000 7fd65dd9b700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600ffd20 0x7fd660101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65ed9d700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6601006b0 0x7fd660101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 -- 172.21.15.64:0/2040541144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6601006b0 msgr2=0x7fd660101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6601006b0 0x7fd660101640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 -- 172.21.15.64:0/2040541144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600ffd20 msgr2=0x7fd660101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600ffd20 0x7fd660101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 -- 172.21.15.64:0/2040541144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6500095f0 con 0x7fd6600feb30 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd65e59c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd650000c00 tx=0x7fd650005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65001d070 con 0x7fd6600feb30 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66019fbc0 con 0x7fd6600feb30 2023-07-14T18:23:27.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd65000be50 con 0x7fd6600feb30 2023-07-14T18:23:27.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65000f460 con 0x7fd6600feb30 2023-07-14T18:23:27.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.448+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66019ffe0 con 0x7fd6600feb30 2023-07-14T18:23:27.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.449+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66010a1b0 con 0x7fd6600feb30 2023-07-14T18:23:27.505 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.450+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd6500226e0 con 0x7fd6600feb30 2023-07-14T18:23:27.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.451+0000 7fd64f7fe700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 0x7fd6480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.451+0000 7fd65dd9b700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 0x7fd6480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.451+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd6500ce520 con 0x7fd6600feb30 2023-07-14T18:23:27.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.451+0000 7fd65dd9b700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 0x7fd6480a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd654009bf0 tx=0x7fd654009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:27.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.454+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd650097b10 con 0x7fd6600feb30 2023-07-14T18:23:27.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.686+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd660069140 con 0x7fd6480a1150 2023-07-14T18:23:27.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.690+0000 7fd64f7fe700 1 -- 172.21.15.64:0/2040541144 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fd660069140 con 0x7fd6480a1150 2023-07-14T18:23:27.742 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/_nogroup/sv_0/4b2cb415-bdae-42ca-a661-4eaba754ca09 2023-07-14T18:23:27.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 msgr2=0x7fd6480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 0x7fd6480a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd654009bf0 tx=0x7fd654009260 comp rx=0 tx=0).stop 2023-07-14T18:23:27.744 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 msgr2=0x7fd660102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fd650000c00 tx=0x7fd650005ef0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 shutdown_connections 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd6601006b0 0x7fd660101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd6480a1150 0x7fd6480a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6600feb30 0x7fd660102ac0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 --2- 172.21.15.64:0/2040541144 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd6600ffd20 0x7fd660101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.745 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 >> 172.21.15.64:0/2040541144 conn(0x7fd6600fa190 msgr2=0x7fd660068a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:27.746 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 shutdown_connections 2023-07-14T18:23:27.746 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.692+0000 7fd664d3c700 1 -- 172.21.15.64:0/2040541144 wait complete. 2023-07-14T18:23:27.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T18:23:27.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 -- 172.21.15.64:0/2657780066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc100f50 msgr2=0x7f17cc1013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 --2- 172.21.15.64:0/2657780066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc100f50 0x7f17cc1013a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009990 tx=0x7f17b4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 -- 172.21.15.64:0/2657780066 shutdown_connections 2023-07-14T18:23:27.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 --2- 172.21.15.64:0/2657780066 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc1018e0 0x7f17cc1081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 --2- 172.21.15.64:0/2657780066 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 0x7f17cc0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 --2- 172.21.15.64:0/2657780066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc100f50 0x7f17cc1013a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 -- 172.21.15.64:0/2657780066 >> 172.21.15.64:0/2657780066 conn(0x7f17cc0f98a0 msgr2=0x7f17cc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 -- 172.21.15.64:0/2657780066 shutdown_connections 2023-07-14T18:23:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.836+0000 7f17d13af700 1 -- 172.21.15.64:0/2657780066 wait complete. 2023-07-14T18:23:27.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 Processor -- start 2023-07-14T18:23:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 -- start start 2023-07-14T18:23:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 0x7f17cc1957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc100f50 0x7f17cc195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc19d350 con 0x7f17cc1018e0 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc19d490 con 0x7f17cc0fdee0 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17d13af700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc19d620 con 0x7f17cc100f50 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47358/0 (socket says 172.21.15.64:47358) 2023-07-14T18:23:27.893 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17cb7fe700 1 -- 172.21.15.64:0/1678920191 learned_addr learned my addr 172.21.15.64:0/1678920191 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17caffd700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 0x7f17cc1957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.894 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.837+0000 7f17ca7fc700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc100f50 0x7f17cc195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 -- 172.21.15.64:0/1678920191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc100f50 msgr2=0x7f17cc195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc100f50 0x7f17cc195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 -- 172.21.15.64:0/1678920191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 msgr2=0x7f17cc1957d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 0x7f17cc1957d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 -- 172.21.15.64:0/1678920191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b40095f0 con 0x7f17cc1018e0 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17cb7fe700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17c4007d70 tx=0x7f17c400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c400ad40 con 0x7f17cc1018e0 2023-07-14T18:23:27.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f17c40128b0 con 0x7f17cc1018e0 2023-07-14T18:23:27.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17cc19d900 con 0x7f17cc1018e0 2023-07-14T18:23:27.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c40115a0 con 0x7f17cc1018e0 2023-07-14T18:23:27.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.838+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17cc19ddc0 con 0x7f17cc1018e0 2023-07-14T18:23:27.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.839+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17cc1072f0 con 0x7f17cc1018e0 2023-07-14T18:23:27.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.840+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f17c4012a20 con 0x7f17cc1018e0 2023-07-14T18:23:27.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.841+0000 7f17c3fff700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 0x7f17b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:27.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.841+0000 7f17caffd700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 0x7f17b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:27.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.841+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f17c40ccdd0 con 0x7f17cc1018e0 2023-07-14T18:23:27.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.841+0000 7f17caffd700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 0x7f17b80a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0674e0 tx=0x7f17bc00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:27.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:27.845+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f17c40962d0 con 0x7f17cc1018e0 2023-07-14T18:23:28.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.076+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f17cc060a80 con 0x7f17b80a1150 2023-07-14T18:23:28.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.103+0000 7f17c3fff700 1 -- 172.21.15.64:0/1678920191 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f17cc060a80 con 0x7f17b80a1150 2023-07-14T18:23:28.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 msgr2=0x7f17b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 0x7f17b80a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f17cc0674e0 tx=0x7f17bc00a3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 msgr2=0x7f17cc19cd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17c4007d70 tx=0x7f17c400a340 comp rx=0 tx=0).stop 2023-07-14T18:23:28.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 shutdown_connections 2023-07-14T18:23:28.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f17b80a1150 0x7f17b80a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17cc1018e0 0x7f17cc19cd80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f17cc0fdee0 0x7f17cc1957d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 --2- 172.21.15.64:0/1678920191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f17cc100f50 0x7f17cc195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 >> 172.21.15.64:0/1678920191 conn(0x7f17cc0f98a0 msgr2=0x7f17cc104430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:28.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 shutdown_connections 2023-07-14T18:23:28.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.105+0000 7f17d13af700 1 -- 172.21.15.64:0/1678920191 wait complete. 2023-07-14T18:23:28.169 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T18:23:28.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 -- 172.21.15.64:0/1075425287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40ffd00 msgr2=0x7fcfe4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 --2- 172.21.15.64:0/1075425287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40ffd00 0x7fcfe4100150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009990 tx=0x7fcfd4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 -- 172.21.15.64:0/1075425287 shutdown_connections 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 --2- 172.21.15.64:0/1075425287 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe4100690 0x7fcfe4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 --2- 172.21.15.64:0/1075425287 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe40feb10 0x7fcfe40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 --2- 172.21.15.64:0/1075425287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40ffd00 0x7fcfe4100150 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.246+0000 7fcfec156700 1 -- 172.21.15.64:0/1075425287 >> 172.21.15.64:0/1075425287 conn(0x7fcfe40fa130 msgr2=0x7fcfe40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:28.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- 172.21.15.64:0/1075425287 shutdown_connections 2023-07-14T18:23:28.300 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- 172.21.15.64:0/1075425287 wait complete. 2023-07-14T18:23:28.301 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 Processor -- start 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- start start 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 0x7fcfe419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 0x7fcfe419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419d410 con 0x7fcfe40feb10 2023-07-14T18:23:28.302 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419d580 con 0x7fcfe4100690 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfec156700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419d740 con 0x7fcfe40ffd00 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.247+0000 7fcfe9ef2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47370/0 (socket says 172.21.15.64:47370) 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 -- 172.21.15.64:0/3964929807 learned_addr learned my addr 172.21.15.64:0/3964929807 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe96f1700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 0x7fcfe419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.303 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfea6f3700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 0x7fcfe419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 -- 172.21.15.64:0/3964929807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 msgr2=0x7fcfe419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 0x7fcfe419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 -- 172.21.15.64:0/3964929807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 msgr2=0x7fcfe419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 0x7fcfe419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 -- 172.21.15.64:0/3964929807 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd40095f0 con 0x7fcfe40feb10 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe96f1700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 0x7fcfe419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:28.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfea6f3700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 0x7fcfe419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfe9ef2700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000c990 tx=0x7fcfe000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0007b00 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcfe0007c60 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe419da20 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe0018660 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.248+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe419df30 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.249+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe4106150 con 0x7fcfe40feb10 2023-07-14T18:23:28.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.250+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcfe000f450 con 0x7fcfe40feb10 2023-07-14T18:23:28.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.251+0000 7fcfdaffd700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 0x7fcfd00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.251+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcfe00ccb80 con 0x7fcfe40feb10 2023-07-14T18:23:28.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.251+0000 7fcfe96f1700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 0x7fcfd00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.251+0000 7fcfe96f1700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 0x7fcfd00a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b5c0 tx=0x7fcfd4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:28.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.255+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcfe0096080 con 0x7fcfe40feb10 2023-07-14T18:23:28.328 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.276+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcfe0096450 con 0x7fcfe40feb10 2023-07-14T18:23:28.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:28.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:active 2023-07-14T18:23:28.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T18:23:28.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:23:28.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: Cluster is now healthy 2023-07-14T18:23:28.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T18:23:28.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T18:23:28.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:28 smithi105 ceph-mon[98013]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-14T18:23:28.562 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.510+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfe4060a80 con 0x7fcfd00a1150 2023-07-14T18:23:28.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.514+0000 7fcfdaffd700 1 -- 172.21.15.64:0/3964929807 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7fcfe4060a80 con 0x7fcfd00a1150 2023-07-14T18:23:28.566 INFO:teuthology.orchestra.run.smithi064.stdout:/volumes/_nogroup/sv_1/ad305f6a-1ca3-4310-81c4-8d2f50074e2e 2023-07-14T18:23:28.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 msgr2=0x7fcfd00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 0x7fcfd00a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b5c0 tx=0x7fcfd4005fb0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 msgr2=0x7fcfe4068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcfe000c990 tx=0x7fcfe000cd50 comp rx=0 tx=0).stop 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 shutdown_connections 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcfd00a1150 0x7fcfd00a3600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfe4100690 0x7fcfe419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe40feb10 0x7fcfe4068310 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 --2- 172.21.15.64:0/3964929807 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcfe40ffd00 0x7fcfe419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 >> 172.21.15.64:0/3964929807 conn(0x7fcfe40fa130 msgr2=0x7fcfe4103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:28.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 shutdown_connections 2023-07-14T18:23:28.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.516+0000 7fcfec156700 1 -- 172.21.15.64:0/3964929807 wait complete. 2023-07-14T18:23:28.579 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:active 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: Cluster is now healthy 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T18:23:28.580 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T18:23:28.581 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:28 smithi064 ceph-mon[96720]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-14T18:23:28.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:23:28.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:28.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] up:active 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: Cluster is now healthy 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=h=up:active,3=c=up:active} 8 up:standby 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2023-07-14T18:23:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:28 smithi144 ceph-mon[98417]: daemon mds.f is now active in filesystem cephfs as rank 4 2023-07-14T18:23:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.655+0000 7f157feb4700 1 -- 172.21.15.64:0/758435199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15780fed40 msgr2=0x7f15780ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.655+0000 7f157feb4700 1 --2- 172.21.15.64:0/758435199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15780fed40 0x7f15780ff1b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1568009990 tx=0x7f1568009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- 172.21.15.64:0/758435199 shutdown_connections 2023-07-14T18:23:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- 172.21.15.64:0/758435199 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1578100180 0x7f1578100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- 172.21.15.64:0/758435199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780ff6f0 0x7f1578106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- 172.21.15.64:0/758435199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15780fed40 0x7f15780ff1b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- 172.21.15.64:0/758435199 >> 172.21.15.64:0/758435199 conn(0x7f15780f98a0 msgr2=0x7f15780fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- 172.21.15.64:0/758435199 shutdown_connections 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- 172.21.15.64:0/758435199 wait complete. 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 Processor -- start 2023-07-14T18:23:28.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- start start 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780fed40 0x7f15781957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15780ff6f0 0x7f1578195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d360 con 0x7f1578100180 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d4a0 con 0x7f15780fed40 2023-07-14T18:23:28.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.656+0000 7f157feb4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d630 con 0x7f15780ff6f0 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47426/0 (socket says 172.21.15.64:47426) 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 -- 172.21.15.64:0/2418973978 learned_addr learned my addr 172.21.15.64:0/2418973978 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157d44f700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15780ff6f0 0x7f1578195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157dc50700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780fed40 0x7f15781957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 -- 172.21.15.64:0/2418973978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15780ff6f0 msgr2=0x7f1578195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15780ff6f0 0x7f1578195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 -- 172.21.15.64:0/2418973978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780fed40 msgr2=0x7f15781957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780fed40 0x7f15781957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 -- 172.21.15.64:0/2418973978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15680095f0 con 0x7f1578100180 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157e451700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1570007ee0 tx=0x7f15700117c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1570004db0 con 0x7f1578100180 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1570009390 con 0x7f1578100180 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1578100fd0 con 0x7f1578100180 2023-07-14T18:23:28.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1570019650 con 0x7f1578100180 2023-07-14T18:23:28.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.657+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1578101490 con 0x7f1578100180 2023-07-14T18:23:28.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.658+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157810a180 con 0x7f1578100180 2023-07-14T18:23:28.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.660+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1570004f10 con 0x7f1578100180 2023-07-14T18:23:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.660+0000 7f156effd700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 0x7f15640a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.660+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f15700d09e0 con 0x7f1578100180 2023-07-14T18:23:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.660+0000 7f157dc50700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 0x7f15640a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.661+0000 7f157dc50700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 0x7f15640a36d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1574005950 tx=0x7f157400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:28.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.664+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1570099de0 con 0x7f1578100180 2023-07-14T18:23:28.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.907+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1578101d80 con 0x7f1578100180 2023-07-14T18:23:28.959 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.907+0000 7f156effd700 1 -- 172.21.15.64:0/2418973978 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7f1570027020 con 0x7f1578100180 2023-07-14T18:23:28.960 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:28.960 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:23:28.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 msgr2=0x7f15640a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 0x7f15640a36d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1574005950 tx=0x7f157400b410 comp rx=0 tx=0).stop 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 msgr2=0x7f157819cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1570007ee0 tx=0x7f15700117c0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 shutdown_connections 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15640a1220 0x7f15640a36d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1578100180 0x7f157819cd90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15780fed40 0x7f15781957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 --2- 172.21.15.64:0/2418973978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15780ff6f0 0x7f1578195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:28.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 >> 172.21.15.64:0/2418973978 conn(0x7f15780f98a0 msgr2=0x7f15780fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:28.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 shutdown_connections 2023-07-14T18:23:28.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:28.910+0000 7f157feb4700 1 -- 172.21.15.64:0/2418973978 wait complete. 2023-07-14T18:23:28.964 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:23:28.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:23:29.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 -- 172.21.15.64:0/3497246415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80fee40 msgr2=0x7f12e80ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 --2- 172.21.15.64:0/3497246415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80fee40 0x7f12e80ff2b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009990 tx=0x7f12dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 -- 172.21.15.64:0/3497246415 shutdown_connections 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 --2- 172.21.15.64:0/3497246415 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80ff7f0 0x7f12e80ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 --2- 172.21.15.64:0/3497246415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 0x7f12e80fe900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 --2- 172.21.15.64:0/3497246415 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80fee40 0x7f12e80ff2b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 -- 172.21.15.64:0/3497246415 >> 172.21.15.64:0/3497246415 conn(0x7f12e80fa100 msgr2=0x7f12e80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 -- 172.21.15.64:0/3497246415 shutdown_connections 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.048+0000 7f12ee080700 1 -- 172.21.15.64:0/3497246415 wait complete. 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 Processor -- start 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 -- start start 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 0x7f12e8104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80fee40 0x7f12e81045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e81013a0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e8101510 con 0x7f12e80fe4f0 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12ee080700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e81016d0 con 0x7f12e80fee40 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47440/0 (socket says 172.21.15.64:47440) 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 -- 172.21.15.64:0/4058403385 learned_addr learned my addr 172.21.15.64:0/4058403385 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:29.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e77fe700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 0x7f12e8104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e6ffd700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80fee40 0x7f12e81045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 -- 172.21.15.64:0/4058403385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80fee40 msgr2=0x7f12e81045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80fee40 0x7f12e81045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 -- 172.21.15.64:0/4058403385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 msgr2=0x7f12e8104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.049+0000 7f12e7fff700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 0x7f12e8104070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12e7fff700 1 -- 172.21.15.64:0/4058403385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc0095f0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12e7fff700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12e0007ee0 tx=0x7f12e000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000ddd0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12e000f040 con 0x7f12e80ff7f0 2023-07-14T18:23:29.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e81a3fa0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00047d0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.050+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e81a43d0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.051+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e81092d0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.052+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f12e0003f20 con 0x7f12e80ff7f0 2023-07-14T18:23:29.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.053+0000 7f12e4ff9700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 0x7f12d00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.053+0000 7f12e77fe700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 0x7f12d00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.053+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f12e00ccdf0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.053+0000 7f12e77fe700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 0x7f12d00a36d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005950 tx=0x7f12d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.056+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12e0096190 con 0x7f12e80ff7f0 2023-07-14T18:23:29.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.259+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e802caf0 con 0x7f12e80ff7f0 2023-07-14T18:23:29.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.261+0000 7f12e4ff9700 1 -- 172.21.15.64:0/4058403385 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f12e0023090 con 0x7f12e80ff7f0 2023-07-14T18:23:29.313 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:29.313 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:29.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 msgr2=0x7f12d00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 0x7f12d00a36d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f12d8005950 tx=0x7f12d800b410 comp rx=0 tx=0).stop 2023-07-14T18:23:29.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 msgr2=0x7f12e8100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f12e0007ee0 tx=0x7f12e000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 shutdown_connections 2023-07-14T18:23:29.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12d00a1220 0x7f12d00a36d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e80ff7f0 0x7f12e8100c00 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12e80fe4f0 0x7f12e8104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 --2- 172.21.15.64:0/4058403385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f12e80fee40 0x7f12e81045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 >> 172.21.15.64:0/4058403385 conn(0x7f12e80fa100 msgr2=0x7f12e80fbdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:29.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 shutdown_connections 2023-07-14T18:23:29.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.263+0000 7f12ee080700 1 -- 172.21.15.64:0/4058403385 wait complete. 2023-07-14T18:23:29.332 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T18:23:29.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T18:23:29.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 -- 172.21.15.64:0/1054450783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ff020 msgr2=0x7fb6d40ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 --2- 172.21.15.64:0/1054450783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ff020 0x7fb6d40ff430 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4009990 tx=0x7fb6c4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 -- 172.21.15.64:0/1054450783 shutdown_connections 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 --2- 172.21.15.64:0/1054450783 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d4106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 --2- 172.21.15.64:0/1054450783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ff020 0x7fb6d40ff430 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 --2- 172.21.15.64:0/1054450783 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ffd30 0x7fb6d41001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 -- 172.21.15.64:0/1054450783 >> 172.21.15.64:0/1054450783 conn(0x7fb6d40f98a0 msgr2=0x7fb6d40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 -- 172.21.15.64:0/1054450783 shutdown_connections 2023-07-14T18:23:29.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.411+0000 7fb6d8990700 1 -- 172.21.15.64:0/1054450783 wait complete. 2023-07-14T18:23:29.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 Processor -- start 2023-07-14T18:23:29.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 -- start start 2023-07-14T18:23:29.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 0x7fb6d40fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d419cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d419d360 con 0x7fb6d40ffd30 2023-07-14T18:23:29.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d419d4d0 con 0x7fb6d41006e0 2023-07-14T18:23:29.468 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d8990700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6d419d690 con 0x7fb6d40ff020 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47460/0 (socket says 172.21.15.64:47460) 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 -- 172.21.15.64:0/2349106877 learned_addr learned my addr 172.21.15.64:0/2349106877 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d2d9d700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d419cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d259c700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 0x7fb6d40fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 -- 172.21.15.64:0/2349106877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 msgr2=0x7fb6d40fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.469 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 0x7fb6d40fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 -- 172.21.15.64:0/2349106877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 msgr2=0x7fb6d419cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d419cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d1d9b700 1 -- 172.21.15.64:0/2349106877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40095f0 con 0x7fb6d40ffd30 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d259c700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 0x7fb6d40fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.412+0000 7fb6d2d9d700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6d1d9b700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d980 tx=0x7fb6bc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc00de60 con 0x7fb6d40ffd30 2023-07-14T18:23:29.470 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb6bc0044b0 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d419d970 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6bc003b40 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.413+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d419de80 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.414+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d410a190 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.415+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb6bc004870 con 0x7fb6d40ffd30 2023-07-14T18:23:29.471 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.415+0000 7fb6cb7fe700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 0x7fb6c00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.416+0000 7fb6d259c700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 0x7fb6c00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.416+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb6bc0cccb0 con 0x7fb6d40ffd30 2023-07-14T18:23:29.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.416+0000 7fb6d259c700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 0x7fb6c00a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40095c0 tx=0x7fb6c4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.474 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.419+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb6bc0960b0 con 0x7fb6d40ffd30 2023-07-14T18:23:29.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: pgmap v266: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2023-07-14T18:23:29.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:23:29.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:active 2023-07-14T18:23:29.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T18:23:29.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:23:29.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2418973978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4058403385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.611+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb6d4060a80 con 0x7fb6c00a1220 2023-07-14T18:23:29.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.627+0000 7fb6cb7fe700 1 -- 172.21.15.64:0/2349106877 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422126 (secure 0 0 0) 0x7fb6d4060a80 con 0x7fb6c00a1220 2023-07-14T18:23:29.686 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:29.688 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: pgmap v266: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:active 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:23:29.689 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2418973978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.690 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:29 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4058403385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 msgr2=0x7fb6c00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 0x7fb6c00a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb6c40095c0 tx=0x7fb6c4009510 comp rx=0 tx=0).stop 2023-07-14T18:23:29.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 msgr2=0x7fb6d419a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb6bc00d980 tx=0x7fb6bc00dd40 comp rx=0 tx=0).stop 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 shutdown_connections 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb6d40ff020 0x7fb6d40fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb6c00a1220 0x7fb6c00a36d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb6d41006e0 0x7fb6d419cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 --2- 172.21.15.64:0/2349106877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6d40ffd30 0x7fb6d419a580 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 >> 172.21.15.64:0/2349106877 conn(0x7fb6d40f98a0 msgr2=0x7fb6d40fbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:29.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 shutdown_connections 2023-07-14T18:23:29.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.631+0000 7fb6d8990700 1 -- 172.21.15.64:0/2349106877 wait complete. 2023-07-14T18:23:29.692 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:23:29.696 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2023-07-14T18:23:29.462402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599993,"num_objects":83,"num_object_clones":0,"num_object_copies":250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":361,"num_write_kb":3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337128,"kb_used_data":12936,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604696,"statfs":{"total":1151940427776,"available":1151595208704,"internally_reserved":0,"allocated":13246464,"data_stored":7246576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5174,"num_objects":59,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":133,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227165+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:23:11.227165+0000","last_peered":"2023-07-14T18:23:11.227165+0000","last_clean":"2023-07-14T18:23:11.227165+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:23:11.227165+0000","last_undegraded":"2023-07-14T18:23:11.227165+0000","last_fullsized":"2023-07-14T18:23:11.227165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:09.995395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226781+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:23:11.226781+0000","last_peered":"2023-07-14T18:23:11.226781+0000","last_clean":"2023-07-14T18:23:11.226781+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:23:11.226781+0000","last_undegraded":"2023-07-14T18:23:11.226781+0000","last_fullsized":"2023-07-14T18:23:11.226781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:58.662673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226533+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:23:11.226533+0000","last_peered":"2023-07-14T18:23:11.226533+0000","last_clean":"2023-07-14T18:23:11.226533+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:23:11.226533+0000","last_undegraded":"2023-07-14T18:23:11.226533+0000","last_fullsized":"2023-07-14T18:23:11.226533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:50.079766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227979+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:23:11.227979+0000","last_peered":"2023-07-14T18:23:11.227979+0000","last_clean":"2023-07-14T18:23:11.227979+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:23:11.227979+0000","last_undegraded":"2023-07-14T18:23:11.227979+0000","last_fullsized":"2023-07-14T18:23:11.227979+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:06.130874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227740+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:23:11.227740+0000","last_peered":"2023-07-14T18:23:11.227740+0000","last_clean":"2023-07-14T18:23:11.227740+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:23:11.227740+0000","last_undegraded":"2023-07-14T18:23:11.227740+0000","last_fullsized":"2023-07-14T18:23:11.227740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:12.302519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227464+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:23:11.227464+0000","last_peered":"2023-07-14T18:23:11.227464+0000","last_clean":"2023-07-14T18:23:11.227464+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:23:11.227464+0000","last_undegraded":"2023-07-14T18:23:11.227464+0000","last_fullsized":"2023-07-14T18:23:11.227464+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:01.500398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610199+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:23:10.610199+0000","last_peered":"2023-07-14T18:23:10.610199+0000","last_clean":"2023-07-14T18:23:10.610199+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:23:10.610199+0000","last_undegraded":"2023-07-14T18:23:10.610199+0000","last_fullsized":"2023-07-14T18:23:10.610199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:21:09.538683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226752+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:23:11.226752+0000","last_peered":"2023-07-14T18:23:11.226752+0000","last_clean":"2023-07-14T18:23:11.226752+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:23:11.226752+0000","last_undegraded":"2023-07-14T18:23:11.226752+0000","last_fullsized":"2023-07-14T18:23:11.226752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:25.294010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.448033+0000","last_change":"2023-07-14T18:23:05.248889+0000","last_active":"2023-07-14T18:23:10.448033+0000","last_peered":"2023-07-14T18:23:10.448033+0000","last_clean":"2023-07-14T18:23:10.448033+0000","last_became_active":"2023-07-14T18:23:05.248500+0000","last_became_peered":"2023-07-14T18:23:05.248500+0000","last_unstale":"2023-07-14T18:23:10.448033+0000","last_undegraded":"2023-07-14T18:23:10.448033+0000","last_fullsized":"2023-07-14T18:23:10.448033+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:53:30.313031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228071+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:23:11.228071+0000","last_peered":"2023-07-14T18:23:11.228071+0000","last_clean":"2023-07-14T18:23:11.228071+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:23:11.228071+0000","last_undegraded":"2023-07-14T18:23:11.228071+0000","last_fullsized":"2023-07-14T18:23:11.228071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:34.155670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610671+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:23:10.610671+0000","last_peered":"2023-07-14T18:23:10.610671+0000","last_clean":"2023-07-14T18:23:10.610671+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:23:10.610671+0000","last_undegraded":"2023-07-14T18:23:10.610671+0000","last_fullsized":"2023-07-14T18:23:10.610671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:08.072924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227886+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:23:11.227886+0000","last_peered":"2023-07-14T18:23:11.227886+0000","last_clean":"2023-07-14T18:23:11.227886+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:23:11.227886+0000","last_undegraded":"2023-07-14T18:23:11.227886+0000","last_fullsized":"2023-07-14T18:23:11.227886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:15.483937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444815+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:23:10.444815+0000","last_peered":"2023-07-14T18:23:10.444815+0000","last_clean":"2023-07-14T18:23:10.444815+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:23:10.444815+0000","last_undegraded":"2023-07-14T18:23:10.444815+0000","last_fullsized":"2023-07-14T18:23:10.444815+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:06.195971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226320+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:23:11.226320+0000","last_peered":"2023-07-14T18:23:11.226320+0000","last_clean":"2023-07-14T18:23:11.226320+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:23:11.226320+0000","last_undegraded":"2023-07-14T18:23:11.226320+0000","last_fullsized":"2023-07-14T18:23:11.226320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:40.019875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226467+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:23:11.226467+0000","last_peered":"2023-07-14T18:23:11.226467+0000","last_clean":"2023-07-14T18:23:11.226467+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:23:11.226467+0000","last_undegraded":"2023-07-14T18:23:11.226467+0000","last_fullsized":"2023-07-14T18:23:11.226467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:46.851772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447060+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:23:10.447060+0000","last_peered":"2023-07-14T18:23:10.447060+0000","last_clean":"2023-07-14T18:23:10.447060+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:23:10.447060+0000","last_undegraded":"2023-07-14T18:23:10.447060+0000","last_fullsized":"2023-07-14T18:23:10.447060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:57.018695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227034+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:23:11.227034+0000","last_peered":"2023-07-14T18:23:11.227034+0000","last_clean":"2023-07-14T18:23:11.227034+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:23:11.227034+0000","last_undegraded":"2023-07-14T18:23:11.227034+0000","last_fullsized":"2023-07-14T18:23:11.227034+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:04.302893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444988+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:23:10.444988+0000","last_peered":"2023-07-14T18:23:10.444988+0000","last_clean":"2023-07-14T18:23:10.444988+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:23:10.444988+0000","last_undegraded":"2023-07-14T18:23:10.444988+0000","last_fullsized":"2023-07-14T18:23:10.444988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:07.815568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445549+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:23:10.445549+0000","last_peered":"2023-07-14T18:23:10.445549+0000","last_clean":"2023-07-14T18:23:10.445549+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:23:10.445549+0000","last_undegraded":"2023-07-14T18:23:10.445549+0000","last_fullsized":"2023-07-14T18:23:10.445549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:26.709513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445503+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:23:10.445503+0000","last_peered":"2023-07-14T18:23:10.445503+0000","last_clean":"2023-07-14T18:23:10.445503+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:23:10.445503+0000","last_undegraded":"2023-07-14T18:23:10.445503+0000","last_fullsized":"2023-07-14T18:23:10.445503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:03.591411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270445+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:23:27.270445+0000","last_peered":"2023-07-14T18:23:27.270445+0000","last_clean":"2023-07-14T18:23:27.270445+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:23:27.270445+0000","last_undegraded":"2023-07-14T18:23:27.270445+0000","last_fullsized":"2023-07-14T18:23:27.270445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:56.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227501+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:23:11.227501+0000","last_peered":"2023-07-14T18:23:11.227501+0000","last_clean":"2023-07-14T18:23:11.227501+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:23:11.227501+0000","last_undegraded":"2023-07-14T18:23:11.227501+0000","last_fullsized":"2023-07-14T18:23:11.227501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:10.900906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226877+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:23:11.226877+0000","last_peered":"2023-07-14T18:23:11.226877+0000","last_clean":"2023-07-14T18:23:11.226877+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:23:11.226877+0000","last_undegraded":"2023-07-14T18:23:11.226877+0000","last_fullsized":"2023-07-14T18:23:11.226877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:59.578019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227969+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:23:11.227969+0000","last_peered":"2023-07-14T18:23:11.227969+0000","last_clean":"2023-07-14T18:23:11.227969+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:23:11.227969+0000","last_undegraded":"2023-07-14T18:23:11.227969+0000","last_fullsized":"2023-07-14T18:23:11.227969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:23.742307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226855+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:23:11.226855+0000","last_peered":"2023-07-14T18:23:11.226855+0000","last_clean":"2023-07-14T18:23:11.226855+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:23:11.226855+0000","last_undegraded":"2023-07-14T18:23:11.226855+0000","last_fullsized":"2023-07-14T18:23:11.226855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:13.892292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227338+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:23:11.227338+0000","last_peered":"2023-07-14T18:23:11.227338+0000","last_clean":"2023-07-14T18:23:11.227338+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:23:11.227338+0000","last_undegraded":"2023-07-14T18:23:11.227338+0000","last_fullsized":"2023-07-14T18:23:11.227338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:36:55.710853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263269+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:23:26.263269+0000","last_peered":"2023-07-14T18:23:26.263269+0000","last_clean":"2023-07-14T18:23:26.263269+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:23:26.263269+0000","last_undegraded":"2023-07-14T18:23:26.263269+0000","last_fullsized":"2023-07-14T18:23:26.263269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:30.855601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447648+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:23:10.447648+0000","last_peered":"2023-07-14T18:23:10.447648+0000","last_clean":"2023-07-14T18:23:10.447648+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:23:10.447648+0000","last_undegraded":"2023-07-14T18:23:10.447648+0000","last_fullsized":"2023-07-14T18:23:10.447648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:35.121798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227220+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:23:11.227220+0000","last_peered":"2023-07-14T18:23:11.227220+0000","last_clean":"2023-07-14T18:23:11.227220+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:23:11.227220+0000","last_undegraded":"2023-07-14T18:23:11.227220+0000","last_fullsized":"2023-07-14T18:23:11.227220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:10.747246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226561+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:23:11.226561+0000","last_peered":"2023-07-14T18:23:11.226561+0000","last_clean":"2023-07-14T18:23:11.226561+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:23:11.226561+0000","last_undegraded":"2023-07-14T18:23:11.226561+0000","last_fullsized":"2023-07-14T18:23:11.226561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:38.727870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226921+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:23:11.226921+0000","last_peered":"2023-07-14T18:23:11.226921+0000","last_clean":"2023-07-14T18:23:11.226921+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:23:11.226921+0000","last_undegraded":"2023-07-14T18:23:11.226921+0000","last_fullsized":"2023-07-14T18:23:11.226921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:28:05.734032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445429+0000","last_change":"2023-07-14T18:23:09.448538+0000","last_active":"2023-07-14T18:23:10.445429+0000","last_peered":"2023-07-14T18:23:10.445429+0000","last_clean":"2023-07-14T18:23:10.445429+0000","last_became_active":"2023-07-14T18:23:09.448108+0000","last_became_peered":"2023-07-14T18:23:09.448108+0000","last_unstale":"2023-07-14T18:23:10.445429+0000","last_undegraded":"2023-07-14T18:23:10.445429+0000","last_fullsized":"2023-07-14T18:23:10.445429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:11.225500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.244900+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:23:24.244900+0000","last_peered":"2023-07-14T18:23:24.244900+0000","last_clean":"2023-07-14T18:23:24.244900+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:23:24.244900+0000","last_undegraded":"2023-07-14T18:23:24.244900+0000","last_fullsized":"2023-07-14T18:23:24.244900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:13.492386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226906+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:23:11.226906+0000","last_peered":"2023-07-14T18:23:11.226906+0000","last_clean":"2023-07-14T18:23:11.226906+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:23:11.226906+0000","last_undegraded":"2023-07-14T18:23:11.226906+0000","last_fullsized":"2023-07-14T18:23:11.226906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:51:56.191875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445464+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:23:10.445464+0000","last_peered":"2023-07-14T18:23:10.445464+0000","last_clean":"2023-07-14T18:23:10.445464+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:23:10.445464+0000","last_undegraded":"2023-07-14T18:23:10.445464+0000","last_fullsized":"2023-07-14T18:23:10.445464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:14.943827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263216+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:23:26.263216+0000","last_peered":"2023-07-14T18:23:26.263216+0000","last_clean":"2023-07-14T18:23:26.263216+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:23:26.263216+0000","last_undegraded":"2023-07-14T18:23:26.263216+0000","last_fullsized":"2023-07-14T18:23:26.263216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:41.156493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229469+0000","last_change":"2023-07-14T18:23:05.254706+0000","last_active":"2023-07-14T18:23:11.229469+0000","last_peered":"2023-07-14T18:23:11.229469+0000","last_clean":"2023-07-14T18:23:11.229469+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.229469+0000","last_undegraded":"2023-07-14T18:23:11.229469+0000","last_fullsized":"2023-07-14T18:23:11.229469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:06:59.136778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227564+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:23:11.227564+0000","last_peered":"2023-07-14T18:23:11.227564+0000","last_clean":"2023-07-14T18:23:11.227564+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:23:11.227564+0000","last_undegraded":"2023-07-14T18:23:11.227564+0000","last_fullsized":"2023-07-14T18:23:11.227564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:44.645790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.254386+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:23:25.254386+0000","last_peered":"2023-07-14T18:23:25.254386+0000","last_clean":"2023-07-14T18:23:25.254386+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:23:25.254386+0000","last_undegraded":"2023-07-14T18:23:25.254386+0000","last_fullsized":"2023-07-14T18:23:25.254386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:35.962587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226991+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:23:11.226991+0000","last_peered":"2023-07-14T18:23:11.226991+0000","last_clean":"2023-07-14T18:23:11.226991+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:23:11.226991+0000","last_undegraded":"2023-07-14T18:23:11.226991+0000","last_fullsized":"2023-07-14T18:23:11.226991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:17.102026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226340+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:23:11.226340+0000","last_peered":"2023-07-14T18:23:11.226340+0000","last_clean":"2023-07-14T18:23:11.226340+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:23:11.226340+0000","last_undegraded":"2023-07-14T18:23:11.226340+0000","last_fullsized":"2023-07-14T18:23:11.226340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:06.464493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.262215+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:23:26.262215+0000","last_peered":"2023-07-14T18:23:26.262215+0000","last_clean":"2023-07-14T18:23:26.262215+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:23:26.262215+0000","last_undegraded":"2023-07-14T18:23:26.262215+0000","last_fullsized":"2023-07-14T18:23:26.262215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:28.512912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444104+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:23:10.444104+0000","last_peered":"2023-07-14T18:23:10.444104+0000","last_clean":"2023-07-14T18:23:10.444104+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:23:10.444104+0000","last_undegraded":"2023-07-14T18:23:10.444104+0000","last_fullsized":"2023-07-14T18:23:10.444104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:14.783284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227144+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:23:11.227144+0000","last_peered":"2023-07-14T18:23:11.227144+0000","last_clean":"2023-07-14T18:23:11.227144+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:23:11.227144+0000","last_undegraded":"2023-07-14T18:23:11.227144+0000","last_fullsized":"2023-07-14T18:23:11.227144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:11:45.259664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.244682+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:23:24.244682+0000","last_peered":"2023-07-14T18:23:24.244682+0000","last_clean":"2023-07-14T18:23:24.244682+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:23:24.244682+0000","last_undegraded":"2023-07-14T18:23:24.244682+0000","last_fullsized":"2023-07-14T18:23:24.244682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:59.026085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447741+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:23:10.447741+0000","last_peered":"2023-07-14T18:23:10.447741+0000","last_clean":"2023-07-14T18:23:10.447741+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:23:10.447741+0000","last_undegraded":"2023-07-14T18:23:10.447741+0000","last_fullsized":"2023-07-14T18:23:10.447741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:46.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.443969+0000","last_change":"2023-07-14T18:23:09.450239+0000","last_active":"2023-07-14T18:23:10.443969+0000","last_peered":"2023-07-14T18:23:10.443969+0000","last_clean":"2023-07-14T18:23:10.443969+0000","last_became_active":"2023-07-14T18:23:09.449827+0000","last_became_peered":"2023-07-14T18:23:09.449827+0000","last_unstale":"2023-07-14T18:23:10.443969+0000","last_undegraded":"2023-07-14T18:23:10.443969+0000","last_fullsized":"2023-07-14T18:23:10.443969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:45:48.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270511+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:23:27.270511+0000","last_peered":"2023-07-14T18:23:27.270511+0000","last_clean":"2023-07-14T18:23:27.270511+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:23:27.270511+0000","last_undegraded":"2023-07-14T18:23:27.270511+0000","last_fullsized":"2023-07-14T18:23:27.270511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:16.477875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226974+0000","last_change":"2023-07-14T18:23:05.255321+0000","last_active":"2023-07-14T18:23:11.226974+0000","last_peered":"2023-07-14T18:23:11.226974+0000","last_clean":"2023-07-14T18:23:11.226974+0000","last_became_active":"2023-07-14T18:23:05.254443+0000","last_became_peered":"2023-07-14T18:23:05.254443+0000","last_unstale":"2023-07-14T18:23:11.226974+0000","last_undegraded":"2023-07-14T18:23:11.226974+0000","last_fullsized":"2023-07-14T18:23:11.226974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:47:37.938549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227625+0000","last_change":"2023-07-14T18:23:09.448642+0000","last_active":"2023-07-14T18:23:11.227625+0000","last_peered":"2023-07-14T18:23:11.227625+0000","last_clean":"2023-07-14T18:23:11.227625+0000","last_became_active":"2023-07-14T18:23:09.448079+0000","last_became_peered":"2023-07-14T18:23:09.448079+0000","last_unstale":"2023-07-14T18:23:11.227625+0000","last_undegraded":"2023-07-14T18:23:11.227625+0000","last_fullsized":"2023-07-14T18:23:11.227625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:25:39.875566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242725+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:23:24.242725+0000","last_peered":"2023-07-14T18:23:24.242725+0000","last_clean":"2023-07-14T18:23:24.242725+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:23:24.242725+0000","last_undegraded":"2023-07-14T18:23:24.242725+0000","last_fullsized":"2023-07-14T18:23:24.242725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:35.880799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447624+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:23:10.447624+0000","last_peered":"2023-07-14T18:23:10.447624+0000","last_clean":"2023-07-14T18:23:10.447624+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:23:10.447624+0000","last_undegraded":"2023-07-14T18:23:10.447624+0000","last_fullsized":"2023-07-14T18:23:10.447624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:01.328580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444552+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:23:10.444552+0000","last_peered":"2023-07-14T18:23:10.444552+0000","last_clean":"2023-07-14T18:23:10.444552+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:23:10.444552+0000","last_undegraded":"2023-07-14T18:23:10.444552+0000","last_fullsized":"2023-07-14T18:23:10.444552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:07:35.036816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'12","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.243600+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:23:24.243600+0000","last_peered":"2023-07-14T18:23:24.243600+0000","last_clean":"2023-07-14T18:23:24.243600+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:23:24.243600+0000","last_undegraded":"2023-07-14T18:23:24.243600+0000","last_fullsized":"2023-07-14T18:23:24.243600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:45.329055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228668+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:23:11.228668+0000","last_peered":"2023-07-14T18:23:11.228668+0000","last_clean":"2023-07-14T18:23:11.228668+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:23:11.228668+0000","last_undegraded":"2023-07-14T18:23:11.228668+0000","last_fullsized":"2023-07-14T18:23:11.228668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:23.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227342+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:23:11.227342+0000","last_peered":"2023-07-14T18:23:11.227342+0000","last_clean":"2023-07-14T18:23:11.227342+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:23:11.227342+0000","last_undegraded":"2023-07-14T18:23:11.227342+0000","last_fullsized":"2023-07-14T18:23:11.227342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:33:09.226720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.269944+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:23:27.269944+0000","last_peered":"2023-07-14T18:23:27.269944+0000","last_clean":"2023-07-14T18:23:27.269944+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:23:27.269944+0000","last_undegraded":"2023-07-14T18:23:27.269944+0000","last_fullsized":"2023-07-14T18:23:27.269944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:15.877531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444561+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:23:10.444561+0000","last_peered":"2023-07-14T18:23:10.444561+0000","last_clean":"2023-07-14T18:23:10.444561+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:23:10.444561+0000","last_undegraded":"2023-07-14T18:23:10.444561+0000","last_fullsized":"2023-07-14T18:23:10.444561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:11:50.717670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226859+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:23:11.226859+0000","last_peered":"2023-07-14T18:23:11.226859+0000","last_clean":"2023-07-14T18:23:11.226859+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:23:11.226859+0000","last_undegraded":"2023-07-14T18:23:11.226859+0000","last_fullsized":"2023-07-14T18:23:11.226859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:40:53.808997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263214+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:23:26.263214+0000","last_peered":"2023-07-14T18:23:26.263214+0000","last_clean":"2023-07-14T18:23:26.263214+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:23:26.263214+0000","last_undegraded":"2023-07-14T18:23:26.263214+0000","last_fullsized":"2023-07-14T18:23:26.263214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:25:36.450416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610444+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:23:10.610444+0000","last_peered":"2023-07-14T18:23:10.610444+0000","last_clean":"2023-07-14T18:23:10.610444+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:23:10.610444+0000","last_undegraded":"2023-07-14T18:23:10.610444+0000","last_fullsized":"2023-07-14T18:23:10.610444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:21.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444636+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:23:10.444636+0000","last_peered":"2023-07-14T18:23:10.444636+0000","last_clean":"2023-07-14T18:23:10.444636+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:23:10.444636+0000","last_undegraded":"2023-07-14T18:23:10.444636+0000","last_fullsized":"2023-07-14T18:23:10.444636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:41.687245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227015+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:23:11.227015+0000","last_peered":"2023-07-14T18:23:11.227015+0000","last_clean":"2023-07-14T18:23:11.227015+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:23:11.227015+0000","last_undegraded":"2023-07-14T18:23:11.227015+0000","last_fullsized":"2023-07-14T18:23:11.227015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:54:03.971193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610150+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:23:10.610150+0000","last_peered":"2023-07-14T18:23:10.610150+0000","last_clean":"2023-07-14T18:23:10.610150+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:23:10.610150+0000","last_undegraded":"2023-07-14T18:23:10.610150+0000","last_fullsized":"2023-07-14T18:23:10.610150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:26.204125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445599+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:23:10.445599+0000","last_peered":"2023-07-14T18:23:10.445599+0000","last_clean":"2023-07-14T18:23:10.445599+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:23:10.445599+0000","last_undegraded":"2023-07-14T18:23:10.445599+0000","last_fullsized":"2023-07-14T18:23:10.445599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:32:00.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'16","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.252676+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:23:25.252676+0000","last_peered":"2023-07-14T18:23:25.252676+0000","last_clean":"2023-07-14T18:23:25.252676+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:23:25.252676+0000","last_undegraded":"2023-07-14T18:23:25.252676+0000","last_fullsized":"2023-07-14T18:23:25.252676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:45.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228441+0000","last_change":"2023-07-14T18:23:05.255683+0000","last_active":"2023-07-14T18:23:11.228441+0000","last_peered":"2023-07-14T18:23:11.228441+0000","last_clean":"2023-07-14T18:23:11.228441+0000","last_became_active":"2023-07-14T18:23:05.255360+0000","last_became_peered":"2023-07-14T18:23:05.255360+0000","last_unstale":"2023-07-14T18:23:11.228441+0000","last_undegraded":"2023-07-14T18:23:11.228441+0000","last_fullsized":"2023-07-14T18:23:11.228441+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:23.880165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228337+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:23:11.228337+0000","last_peered":"2023-07-14T18:23:11.228337+0000","last_clean":"2023-07-14T18:23:11.228337+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:23:11.228337+0000","last_undegraded":"2023-07-14T18:23:11.228337+0000","last_fullsized":"2023-07-14T18:23:11.228337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:18.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226684+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:23:11.226684+0000","last_peered":"2023-07-14T18:23:11.226684+0000","last_clean":"2023-07-14T18:23:11.226684+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:23:11.226684+0000","last_undegraded":"2023-07-14T18:23:11.226684+0000","last_fullsized":"2023-07-14T18:23:11.226684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:33.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227454+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:23:11.227454+0000","last_peered":"2023-07-14T18:23:11.227454+0000","last_clean":"2023-07-14T18:23:11.227454+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.227454+0000","last_undegraded":"2023-07-14T18:23:11.227454+0000","last_fullsized":"2023-07-14T18:23:11.227454+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:55.074433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226809+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:23:11.226809+0000","last_peered":"2023-07-14T18:23:11.226809+0000","last_clean":"2023-07-14T18:23:11.226809+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:23:11.226809+0000","last_undegraded":"2023-07-14T18:23:11.226809+0000","last_fullsized":"2023-07-14T18:23:11.226809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:44.909597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444437+0000","last_change":"2023-07-14T18:23:04.418035+0000","last_active":"2023-07-14T18:23:10.444437+0000","last_peered":"2023-07-14T18:23:10.444437+0000","last_clean":"2023-07-14T18:23:10.444437+0000","last_became_active":"2023-07-14T18:23:04.417645+0000","last_became_peered":"2023-07-14T18:23:04.417645+0000","last_unstale":"2023-07-14T18:23:10.444437+0000","last_undegraded":"2023-07-14T18:23:10.444437+0000","last_fullsized":"2023-07-14T18:23:10.444437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:08.545074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227920+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:23:11.227920+0000","last_peered":"2023-07-14T18:23:11.227920+0000","last_clean":"2023-07-14T18:23:11.227920+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:23:11.227920+0000","last_undegraded":"2023-07-14T18:23:11.227920+0000","last_fullsized":"2023-07-14T18:23:11.227920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:47.382928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445312+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:23:10.445312+0000","last_peered":"2023-07-14T18:23:10.445312+0000","last_clean":"2023-07-14T18:23:10.445312+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:23:10.445312+0000","last_undegraded":"2023-07-14T18:23:10.445312+0000","last_fullsized":"2023-07-14T18:23:10.445312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:36:47.920949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.290262+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:23:28.290262+0000","last_peered":"2023-07-14T18:23:28.290262+0000","last_clean":"2023-07-14T18:23:28.290262+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:23:28.290262+0000","last_undegraded":"2023-07-14T18:23:28.290262+0000","last_fullsized":"2023-07-14T18:23:28.290262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:22:04.367333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227207+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:23:11.227207+0000","last_peered":"2023-07-14T18:23:11.227207+0000","last_clean":"2023-07-14T18:23:11.227207+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:23:11.227207+0000","last_undegraded":"2023-07-14T18:23:11.227207+0000","last_fullsized":"2023-07-14T18:23:11.227207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:56:42.916396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445652+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:23:10.445652+0000","last_peered":"2023-07-14T18:23:10.445652+0000","last_clean":"2023-07-14T18:23:10.445652+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:23:10.445652+0000","last_undegraded":"2023-07-14T18:23:10.445652+0000","last_fullsized":"2023-07-14T18:23:10.445652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:51:24.527443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228927+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:23:11.228927+0000","last_peered":"2023-07-14T18:23:11.228927+0000","last_clean":"2023-07-14T18:23:11.228927+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:23:11.228927+0000","last_undegraded":"2023-07-14T18:23:11.228927+0000","last_fullsized":"2023-07-14T18:23:11.228927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:15.545609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.287546+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:23:28.287546+0000","last_peered":"2023-07-14T18:23:28.287546+0000","last_clean":"2023-07-14T18:23:28.287546+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:23:28.287546+0000","last_undegraded":"2023-07-14T18:23:28.287546+0000","last_fullsized":"2023-07-14T18:23:28.287546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:33:45.754899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610640+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:23:10.610640+0000","last_peered":"2023-07-14T18:23:10.610640+0000","last_clean":"2023-07-14T18:23:10.610640+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:23:10.610640+0000","last_undegraded":"2023-07-14T18:23:10.610640+0000","last_fullsized":"2023-07-14T18:23:10.610640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:17.385324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226985+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:23:11.226985+0000","last_peered":"2023-07-14T18:23:11.226985+0000","last_clean":"2023-07-14T18:23:11.226985+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:23:11.226985+0000","last_undegraded":"2023-07-14T18:23:11.226985+0000","last_fullsized":"2023-07-14T18:23:11.226985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:48.279521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242903+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:23:24.242903+0000","last_peered":"2023-07-14T18:23:24.242903+0000","last_clean":"2023-07-14T18:23:24.242903+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:23:24.242903+0000","last_undegraded":"2023-07-14T18:23:24.242903+0000","last_fullsized":"2023-07-14T18:23:24.242903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:52.919560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.609987+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:23:10.609987+0000","last_peered":"2023-07-14T18:23:10.609987+0000","last_clean":"2023-07-14T18:23:10.609987+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:23:10.609987+0000","last_undegraded":"2023-07-14T18:23:10.609987+0000","last_fullsized":"2023-07-14T18:23:10.609987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:43.598474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227359+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:23:11.227359+0000","last_peered":"2023-07-14T18:23:11.227359+0000","last_clean":"2023-07-14T18:23:11.227359+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:23:11.227359+0000","last_undegraded":"2023-07-14T18:23:11.227359+0000","last_fullsized":"2023-07-14T18:23:11.227359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:34:29.533657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444959+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:23:10.444959+0000","last_peered":"2023-07-14T18:23:10.444959+0000","last_clean":"2023-07-14T18:23:10.444959+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:23:10.444959+0000","last_undegraded":"2023-07-14T18:23:10.444959+0000","last_fullsized":"2023-07-14T18:23:10.444959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:08.456204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.252048+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:23:25.252048+0000","last_peered":"2023-07-14T18:23:25.252048+0000","last_clean":"2023-07-14T18:23:25.252048+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:23:25.252048+0000","last_undegraded":"2023-07-14T18:23:25.252048+0000","last_fullsized":"2023-07-14T18:23:25.252048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:57.271343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226827+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:23:11.226827+0000","last_peered":"2023-07-14T18:23:11.226827+0000","last_clean":"2023-07-14T18:23:11.226827+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:23:11.226827+0000","last_undegraded":"2023-07-14T18:23:11.226827+0000","last_fullsized":"2023-07-14T18:23:11.226827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:41.538513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'3","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263857+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:23:26.263857+0000","last_peered":"2023-07-14T18:23:26.263857+0000","last_clean":"2023-07-14T18:23:26.263857+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:23:26.263857+0000","last_undegraded":"2023-07-14T18:23:26.263857+0000","last_fullsized":"2023-07-14T18:23:26.263857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:29.140961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445507+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:23:10.445507+0000","last_peered":"2023-07-14T18:23:10.445507+0000","last_clean":"2023-07-14T18:23:10.445507+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:23:10.445507+0000","last_undegraded":"2023-07-14T18:23:10.445507+0000","last_fullsized":"2023-07-14T18:23:10.445507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:01.714171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226933+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:23:11.226933+0000","last_peered":"2023-07-14T18:23:11.226933+0000","last_clean":"2023-07-14T18:23:11.226933+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:23:11.226933+0000","last_undegraded":"2023-07-14T18:23:11.226933+0000","last_fullsized":"2023-07-14T18:23:11.226933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:06:06.174345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263560+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:23:26.263560+0000","last_peered":"2023-07-14T18:23:26.263560+0000","last_clean":"2023-07-14T18:23:26.263560+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:23:26.263560+0000","last_undegraded":"2023-07-14T18:23:26.263560+0000","last_fullsized":"2023-07-14T18:23:26.263560+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:38.063154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227450+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:23:11.227450+0000","last_peered":"2023-07-14T18:23:11.227450+0000","last_clean":"2023-07-14T18:23:11.227450+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:23:11.227450+0000","last_undegraded":"2023-07-14T18:23:11.227450+0000","last_fullsized":"2023-07-14T18:23:11.227450+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:39.395014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445040+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:23:10.445040+0000","last_peered":"2023-07-14T18:23:10.445040+0000","last_clean":"2023-07-14T18:23:10.445040+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:23:10.445040+0000","last_undegraded":"2023-07-14T18:23:10.445040+0000","last_fullsized":"2023-07-14T18:23:10.445040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:54.870365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.243023+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:23:24.243023+0000","last_peered":"2023-07-14T18:23:24.243023+0000","last_clean":"2023-07-14T18:23:24.243023+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:23:24.243023+0000","last_undegraded":"2023-07-14T18:23:24.243023+0000","last_fullsized":"2023-07-14T18:23:24.243023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:54:38.614796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226909+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:23:11.226909+0000","last_peered":"2023-07-14T18:23:11.226909+0000","last_clean":"2023-07-14T18:23:11.226909+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:23:11.226909+0000","last_undegraded":"2023-07-14T18:23:11.226909+0000","last_fullsized":"2023-07-14T18:23:11.226909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:43.634763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227411+0000","last_change":"2023-07-14T18:23:09.449832+0000","last_active":"2023-07-14T18:23:11.227411+0000","last_peered":"2023-07-14T18:23:11.227411+0000","last_clean":"2023-07-14T18:23:11.227411+0000","last_became_active":"2023-07-14T18:23:09.448997+0000","last_became_peered":"2023-07-14T18:23:09.448997+0000","last_unstale":"2023-07-14T18:23:11.227411+0000","last_undegraded":"2023-07-14T18:23:11.227411+0000","last_fullsized":"2023-07-14T18:23:11.227411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:56.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242161+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:23:24.242161+0000","last_peered":"2023-07-14T18:23:24.242161+0000","last_clean":"2023-07-14T18:23:24.242161+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:23:24.242161+0000","last_undegraded":"2023-07-14T18:23:24.242161+0000","last_fullsized":"2023-07-14T18:23:24.242161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:46.321986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226403+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:23:11.226403+0000","last_peered":"2023-07-14T18:23:11.226403+0000","last_clean":"2023-07-14T18:23:11.226403+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:23:11.226403+0000","last_undegraded":"2023-07-14T18:23:11.226403+0000","last_fullsized":"2023-07-14T18:23:11.226403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:03:27.653641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444903+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:23:10.444903+0000","last_peered":"2023-07-14T18:23:10.444903+0000","last_clean":"2023-07-14T18:23:10.444903+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:23:10.444903+0000","last_undegraded":"2023-07-14T18:23:10.444903+0000","last_fullsized":"2023-07-14T18:23:10.444903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:40.273437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268661+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:23:27.268661+0000","last_peered":"2023-07-14T18:23:27.268661+0000","last_clean":"2023-07-14T18:23:27.268661+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:23:27.268661+0000","last_undegraded":"2023-07-14T18:23:27.268661+0000","last_fullsized":"2023-07-14T18:23:27.268661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:25.573588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229498+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:23:11.229498+0000","last_peered":"2023-07-14T18:23:11.229498+0000","last_clean":"2023-07-14T18:23:11.229498+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:23:11.229498+0000","last_undegraded":"2023-07-14T18:23:11.229498+0000","last_fullsized":"2023-07-14T18:23:11.229498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:55.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227675+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:23:11.227675+0000","last_peered":"2023-07-14T18:23:11.227675+0000","last_clean":"2023-07-14T18:23:11.227675+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:23:11.227675+0000","last_undegraded":"2023-07-14T18:23:11.227675+0000","last_fullsized":"2023-07-14T18:23:11.227675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:26:45.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445140+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:23:10.445140+0000","last_peered":"2023-07-14T18:23:10.445140+0000","last_clean":"2023-07-14T18:23:10.445140+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:23:10.445140+0000","last_undegraded":"2023-07-14T18:23:10.445140+0000","last_fullsized":"2023-07-14T18:23:10.445140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:57:27.449607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227548+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:23:11.227548+0000","last_peered":"2023-07-14T18:23:11.227548+0000","last_clean":"2023-07-14T18:23:11.227548+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:23:11.227548+0000","last_undegraded":"2023-07-14T18:23:11.227548+0000","last_fullsized":"2023-07-14T18:23:11.227548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:58.508966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227371+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:23:11.227371+0000","last_peered":"2023-07-14T18:23:11.227371+0000","last_clean":"2023-07-14T18:23:11.227371+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:23:11.227371+0000","last_undegraded":"2023-07-14T18:23:11.227371+0000","last_fullsized":"2023-07-14T18:23:11.227371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:20.402932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.241780+0000","last_change":"2023-07-14T18:23:04.424951+0000","last_active":"2023-07-14T18:23:24.241780+0000","last_peered":"2023-07-14T18:23:24.241780+0000","last_clean":"2023-07-14T18:23:24.241780+0000","last_became_active":"2023-07-14T18:23:04.424641+0000","last_became_peered":"2023-07-14T18:23:04.424641+0000","last_unstale":"2023-07-14T18:23:24.241780+0000","last_undegraded":"2023-07-14T18:23:24.241780+0000","last_fullsized":"2023-07-14T18:23:24.241780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:25.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610293+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:23:10.610293+0000","last_peered":"2023-07-14T18:23:10.610293+0000","last_clean":"2023-07-14T18:23:10.610293+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:23:10.610293+0000","last_undegraded":"2023-07-14T18:23:10.610293+0000","last_fullsized":"2023-07-14T18:23:10.610293+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:11.675787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447265+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:23:10.447265+0000","last_peered":"2023-07-14T18:23:10.447265+0000","last_clean":"2023-07-14T18:23:10.447265+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:23:10.447265+0000","last_undegraded":"2023-07-14T18:23:10.447265+0000","last_fullsized":"2023-07-14T18:23:10.447265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:43.737345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.265288+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:23:26.265288+0000","last_peered":"2023-07-14T18:23:26.265288+0000","last_clean":"2023-07-14T18:23:26.265288+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:23:26.265288+0000","last_undegraded":"2023-07-14T18:23:26.265288+0000","last_fullsized":"2023-07-14T18:23:26.265288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:04.272902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610785+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:23:10.610785+0000","last_peered":"2023-07-14T18:23:10.610785+0000","last_clean":"2023-07-14T18:23:10.610785+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:23:10.610785+0000","last_undegraded":"2023-07-14T18:23:10.610785+0000","last_fullsized":"2023-07-14T18:23:10.610785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:27:06.003284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226097+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:23:11.226097+0000","last_peered":"2023-07-14T18:23:11.226097+0000","last_clean":"2023-07-14T18:23:11.226097+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:23:11.226097+0000","last_undegraded":"2023-07-14T18:23:11.226097+0000","last_fullsized":"2023-07-14T18:23:11.226097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:26.374670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263332+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:23:26.263332+0000","last_peered":"2023-07-14T18:23:26.263332+0000","last_clean":"2023-07-14T18:23:26.263332+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:23:26.263332+0000","last_undegraded":"2023-07-14T18:23:26.263332+0000","last_fullsized":"2023-07-14T18:23:26.263332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:05:36.820471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610456+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:23:10.610456+0000","last_peered":"2023-07-14T18:23:10.610456+0000","last_clean":"2023-07-14T18:23:10.610456+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:23:10.610456+0000","last_undegraded":"2023-07-14T18:23:10.610456+0000","last_fullsized":"2023-07-14T18:23:10.610456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:55:16.680648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228516+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:23:11.228516+0000","last_peered":"2023-07-14T18:23:11.228516+0000","last_clean":"2023-07-14T18:23:11.228516+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:23:11.228516+0000","last_undegraded":"2023-07-14T18:23:11.228516+0000","last_fullsized":"2023-07-14T18:23:11.228516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:45.616005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268218+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:23:27.268218+0000","last_peered":"2023-07-14T18:23:27.268218+0000","last_clean":"2023-07-14T18:23:27.268218+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:23:27.268218+0000","last_undegraded":"2023-07-14T18:23:27.268218+0000","last_fullsized":"2023-07-14T18:23:27.268218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:35.670294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.611027+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:23:10.611027+0000","last_peered":"2023-07-14T18:23:10.611027+0000","last_clean":"2023-07-14T18:23:10.611027+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:23:10.611027+0000","last_undegraded":"2023-07-14T18:23:10.611027+0000","last_fullsized":"2023-07-14T18:23:10.611027+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:51.422332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229684+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:23:11.229684+0000","last_peered":"2023-07-14T18:23:11.229684+0000","last_clean":"2023-07-14T18:23:11.229684+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:23:11.229684+0000","last_undegraded":"2023-07-14T18:23:11.229684+0000","last_fullsized":"2023-07-14T18:23:11.229684+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:17.955330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445263+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:23:10.445263+0000","last_peered":"2023-07-14T18:23:10.445263+0000","last_clean":"2023-07-14T18:23:10.445263+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:23:10.445263+0000","last_undegraded":"2023-07-14T18:23:10.445263+0000","last_fullsized":"2023-07-14T18:23:10.445263+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:47.687598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445138+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:23:10.445138+0000","last_peered":"2023-07-14T18:23:10.445138+0000","last_clean":"2023-07-14T18:23:10.445138+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:23:10.445138+0000","last_undegraded":"2023-07-14T18:23:10.445138+0000","last_fullsized":"2023-07-14T18:23:10.445138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:22.609183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226339+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:23:11.226339+0000","last_peered":"2023-07-14T18:23:11.226339+0000","last_clean":"2023-07-14T18:23:11.226339+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:23:11.226339+0000","last_undegraded":"2023-07-14T18:23:11.226339+0000","last_fullsized":"2023-07-14T18:23:11.226339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:45.583600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445397+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:23:10.445397+0000","last_peered":"2023-07-14T18:23:10.445397+0000","last_clean":"2023-07-14T18:23:10.445397+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:23:10.445397+0000","last_undegraded":"2023-07-14T18:23:10.445397+0000","last_fullsized":"2023-07-14T18:23:10.445397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:10.666041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227616+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:23:11.227616+0000","last_peered":"2023-07-14T18:23:11.227616+0000","last_clean":"2023-07-14T18:23:11.227616+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:23:11.227616+0000","last_undegraded":"2023-07-14T18:23:11.227616+0000","last_fullsized":"2023-07-14T18:23:11.227616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:50.867762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445702+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:23:10.445702+0000","last_peered":"2023-07-14T18:23:10.445702+0000","last_clean":"2023-07-14T18:23:10.445702+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:23:10.445702+0000","last_undegraded":"2023-07-14T18:23:10.445702+0000","last_fullsized":"2023-07-14T18:23:10.445702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:50.705252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227634+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:23:11.227634+0000","last_peered":"2023-07-14T18:23:11.227634+0000","last_clean":"2023-07-14T18:23:11.227634+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:23:11.227634+0000","last_undegraded":"2023-07-14T18:23:11.227634+0000","last_fullsized":"2023-07-14T18:23:11.227634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:57.010106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228805+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:23:11.228805+0000","last_peered":"2023-07-14T18:23:11.228805+0000","last_clean":"2023-07-14T18:23:11.228805+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:23:11.228805+0000","last_undegraded":"2023-07-14T18:23:11.228805+0000","last_fullsized":"2023-07-14T18:23:11.228805+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:27.226698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227229+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:23:11.227229+0000","last_peered":"2023-07-14T18:23:11.227229+0000","last_clean":"2023-07-14T18:23:11.227229+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:23:11.227229+0000","last_undegraded":"2023-07-14T18:23:11.227229+0000","last_fullsized":"2023-07-14T18:23:11.227229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:50.246761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447443+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:23:10.447443+0000","last_peered":"2023-07-14T18:23:10.447443+0000","last_clean":"2023-07-14T18:23:10.447443+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:23:10.447443+0000","last_undegraded":"2023-07-14T18:23:10.447443+0000","last_fullsized":"2023-07-14T18:23:10.447443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:22.891172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444900+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:23:10.444900+0000","last_peered":"2023-07-14T18:23:10.444900+0000","last_clean":"2023-07-14T18:23:10.444900+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:23:10.444900+0000","last_undegraded":"2023-07-14T18:23:10.444900+0000","last_fullsized":"2023-07-14T18:23:10.444900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:52.929194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227023+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:23:11.227023+0000","last_peered":"2023-07-14T18:23:11.227023+0000","last_clean":"2023-07-14T18:23:11.227023+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:23:11.227023+0000","last_undegraded":"2023-07-14T18:23:11.227023+0000","last_fullsized":"2023-07-14T18:23:11.227023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:19.793596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9502,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":179,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":45621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514511,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":458554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.497},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.745},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.78400000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.879}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.84099999999999997}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1048740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.73099999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.872}]},{"osd":6,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.92800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.93799999999999994}]},{"osd":9,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.83599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]}]},{"osd":9,"up_from":64,"seq":274877906968,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":458142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65700000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.38700000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.625}]},{"osd":5,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.48299999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.64500000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.747}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":453662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.433}]},{"osd":3,"last update":"Fri Jul 14 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.82599999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.62},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.65000000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.745},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.754}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.88400000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1054599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40500000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.748},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.751,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.502,"15min":0.497},"min":{"1min":0.313,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.81799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.51100000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":1.02,"15min":1.02},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.99099999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.97099999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":453325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.46800000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.62,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94399999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.748}]},{"osd":3,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.441},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.505,"15min":0.505},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39500000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91600000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86799999999999999,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68100000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85999999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.61699999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.505,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.68600000000000005,"15min":0.67600000000000005},"min":{"1min":0.50900000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.72299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":456718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.505,"15min":0.501},"min":{"1min":0.39100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86899999999999999,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.54800000000000004,"15min":0.52700000000000002},"min":{"1min":0.434,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.443,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64000000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.71499999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69999999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.58999999999999997,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.40100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.76000000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.86399999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.84599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":453345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.54000000000000004,"15min":0.496},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54800000000000004,"15min":0.51100000000000001},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63}]},{"osd":2,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89500000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.502},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88900000000000001,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Jul 14 18:21:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.39600000000000002,"15min":0.38200000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26400000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.439}]},{"osd":4,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.59299999999999997,"15min":0.56299999999999994},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.47599999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.44600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.745}]},{"osd":8,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59899999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.66700000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.85899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":458594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.375},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52800000000000002},"min":{"1min":0.374,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:22:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.47799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57299999999999995,"15min":0.55200000000000005},"min":{"1min":0.505,"5min":0.38,"15min":0.38},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.502}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.753},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.78200000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.76800000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.80600000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":453073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.27300000000000002,"5min":0.161,"15min":0.161},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.38100000000000001,"15min":0.35499999999999998},"min":{"1min":0.30599999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.57799999999999996,"15min":0.55100000000000005},"min":{"1min":0.41399999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.58699999999999997,"15min":0.56399999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.80900000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.58999999999999997,"15min":0.56499999999999995},"min":{"1min":0.45100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.91700000000000004,"5min":1.274,"15min":1.274},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58699999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.125,"5min":1.22,"15min":1.22},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.84199999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.754}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]}]},{"osd":4,"up_from":33,"seq":141733920813,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40999999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.62},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.40899999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.70099999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.89300000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71799999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.79500000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.75}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1045151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.437}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.65100000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.61899999999999999,"15min":0.60499999999999998},"min":{"1min":0.48399999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.53200000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.77700000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.70399999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.72799999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":23831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:23:29.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T18:23:29.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: pgmap v266: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 4 op/s 2023-07-14T18:23:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T18:23:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] up:active 2023-07-14T18:23:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: fsmap cephfs:5 7 up:standby 5 up:active 2023-07-14T18:23:29.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:23:29.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2418973978' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:29 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4058403385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/3670297413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d181006d0 msgr2=0x7f6d18106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/3670297413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d181006d0 0x7f6d18106f90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6d1000b0a0 tx=0x7f6d1000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/3670297413 shutdown_connections 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/3670297413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d181006d0 0x7f6d18106f90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/3670297413 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180feb50 0x7f6d180fef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/3670297413 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d180ffd40 0x7f6d18100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/3670297413 >> 172.21.15.64:0/3670297413 conn(0x7f6d180fa150 msgr2=0x7f6d180fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/3670297413 shutdown_connections 2023-07-14T18:23:29.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.809+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/3670297413 wait complete. 2023-07-14T18:23:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 Processor -- start 2023-07-14T18:23:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 -- start start 2023-07-14T18:23:29.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 0x7f6d18195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d181006d0 0x7f6d1819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1819d500 con 0x7f6d180feb50 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1819d640 con 0x7f6d180ffd40 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d1d9f9700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1819d7d0 con 0x7f6d181006d0 2023-07-14T18:23:29.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d16ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d16ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47472/0 (socket says 172.21.15.64:47472) 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d16ffd700 1 -- 172.21.15.64:0/502936524 learned_addr learned my addr 172.21.15.64:0/502936524 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d177fe700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d181006d0 0x7f6d1819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.810+0000 7f6d167fc700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 0x7f6d18195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 -- 172.21.15.64:0/502936524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d181006d0 msgr2=0x7f6d1819cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d181006d0 0x7f6d1819cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 -- 172.21.15.64:0/502936524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 msgr2=0x7f6d18195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 0x7f6d18195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 -- 172.21.15.64:0/502936524 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d10009d00 con 0x7f6d180feb50 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d167fc700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 0x7f6d18195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d16ffd700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e970 tx=0x7f6d0800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08009790 con 0x7f6d180feb50 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1819dab0 con 0x7f6d180feb50 2023-07-14T18:23:29.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6d0800eec0 con 0x7f6d180feb50 2023-07-14T18:23:29.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d08003c10 con 0x7f6d180feb50 2023-07-14T18:23:29.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.811+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1819dfc0 con 0x7f6d180feb50 2023-07-14T18:23:29.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.812+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1810a1e0 con 0x7f6d180feb50 2023-07-14T18:23:29.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.813+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6d08003d90 con 0x7f6d180feb50 2023-07-14T18:23:29.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.813+0000 7f6d1c9f7700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 0x7f6d000a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:29.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.814+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6d08014070 con 0x7f6d180feb50 2023-07-14T18:23:29.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.814+0000 7f6d167fc700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 0x7f6d000a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:29.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.814+0000 7f6d167fc700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 0x7f6d000a36d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009cb0 tx=0x7f6d0c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:29.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:29.817+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6d08096470 con 0x7f6d180feb50 2023-07-14T18:23:30.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.014+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6d18069190 con 0x7f6d000a1220 2023-07-14T18:23:30.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.029+0000 7f6d1c9f7700 1 -- 172.21.15.64:0/502936524 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+422126 (secure 0 0 0) 0x7f6d18069190 con 0x7f6d000a1220 2023-07-14T18:23:30.084 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:30.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 msgr2=0x7f6d000a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:30.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 0x7f6d000a36d0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f6d0c009cb0 tx=0x7f6d0c009260 comp rx=0 tx=0).stop 2023-07-14T18:23:30.088 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 msgr2=0x7f6d18195980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:30.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800e970 tx=0x7f6d0800ec80 comp rx=0 tx=0).stop 2023-07-14T18:23:30.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 shutdown_connections 2023-07-14T18:23:30.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d181006d0 0x7f6d1819cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:30.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d000a1220 0x7f6d000a36d0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:30.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d180feb50 0x7f6d18195980 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:30.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 --2- 172.21.15.64:0/502936524 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d180ffd40 0x7f6d18195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:30.090 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 >> 172.21.15.64:0/502936524 conn(0x7f6d180fa150 msgr2=0x7f6d18068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:30.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 shutdown_connections 2023-07-14T18:23:30.091 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:30.033+0000 7f6d1d9f9700 1 -- 172.21.15.64:0/502936524 wait complete. 2023-07-14T18:23:30.091 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:23:30.097 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":267,"stamp":"2023-07-14T18:23:29.462402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":599993,"num_objects":83,"num_object_clones":0,"num_object_copies":250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":141,"num_read_kb":123,"num_write":361,"num_write_kb":3002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":296,"ondisk_log_size":296,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337128,"kb_used_data":12936,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604696,"statfs":{"total":1151940427776,"available":1151595208704,"internally_reserved":0,"allocated":13246464,"data_stored":7246576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5174,"num_objects":59,"num_object_clones":0,"num_object_copies":178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":133,"num_write_kb":47,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227165+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:23:11.227165+0000","last_peered":"2023-07-14T18:23:11.227165+0000","last_clean":"2023-07-14T18:23:11.227165+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:23:11.227165+0000","last_undegraded":"2023-07-14T18:23:11.227165+0000","last_fullsized":"2023-07-14T18:23:11.227165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:09.995395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226781+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:23:11.226781+0000","last_peered":"2023-07-14T18:23:11.226781+0000","last_clean":"2023-07-14T18:23:11.226781+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:23:11.226781+0000","last_undegraded":"2023-07-14T18:23:11.226781+0000","last_fullsized":"2023-07-14T18:23:11.226781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:58.662673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226533+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:23:11.226533+0000","last_peered":"2023-07-14T18:23:11.226533+0000","last_clean":"2023-07-14T18:23:11.226533+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:23:11.226533+0000","last_undegraded":"2023-07-14T18:23:11.226533+0000","last_fullsized":"2023-07-14T18:23:11.226533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:50.079766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227979+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:23:11.227979+0000","last_peered":"2023-07-14T18:23:11.227979+0000","last_clean":"2023-07-14T18:23:11.227979+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:23:11.227979+0000","last_undegraded":"2023-07-14T18:23:11.227979+0000","last_fullsized":"2023-07-14T18:23:11.227979+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:06.130874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227740+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:23:11.227740+0000","last_peered":"2023-07-14T18:23:11.227740+0000","last_clean":"2023-07-14T18:23:11.227740+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:23:11.227740+0000","last_undegraded":"2023-07-14T18:23:11.227740+0000","last_fullsized":"2023-07-14T18:23:11.227740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:12.302519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227464+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:23:11.227464+0000","last_peered":"2023-07-14T18:23:11.227464+0000","last_clean":"2023-07-14T18:23:11.227464+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:23:11.227464+0000","last_undegraded":"2023-07-14T18:23:11.227464+0000","last_fullsized":"2023-07-14T18:23:11.227464+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:01.500398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610199+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:23:10.610199+0000","last_peered":"2023-07-14T18:23:10.610199+0000","last_clean":"2023-07-14T18:23:10.610199+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:23:10.610199+0000","last_undegraded":"2023-07-14T18:23:10.610199+0000","last_fullsized":"2023-07-14T18:23:10.610199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:21:09.538683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226752+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:23:11.226752+0000","last_peered":"2023-07-14T18:23:11.226752+0000","last_clean":"2023-07-14T18:23:11.226752+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:23:11.226752+0000","last_undegraded":"2023-07-14T18:23:11.226752+0000","last_fullsized":"2023-07-14T18:23:11.226752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:25.294010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.448033+0000","last_change":"2023-07-14T18:23:05.248889+0000","last_active":"2023-07-14T18:23:10.448033+0000","last_peered":"2023-07-14T18:23:10.448033+0000","last_clean":"2023-07-14T18:23:10.448033+0000","last_became_active":"2023-07-14T18:23:05.248500+0000","last_became_peered":"2023-07-14T18:23:05.248500+0000","last_unstale":"2023-07-14T18:23:10.448033+0000","last_undegraded":"2023-07-14T18:23:10.448033+0000","last_fullsized":"2023-07-14T18:23:10.448033+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:53:30.313031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228071+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:23:11.228071+0000","last_peered":"2023-07-14T18:23:11.228071+0000","last_clean":"2023-07-14T18:23:11.228071+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:23:11.228071+0000","last_undegraded":"2023-07-14T18:23:11.228071+0000","last_fullsized":"2023-07-14T18:23:11.228071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:34.155670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610671+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:23:10.610671+0000","last_peered":"2023-07-14T18:23:10.610671+0000","last_clean":"2023-07-14T18:23:10.610671+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:23:10.610671+0000","last_undegraded":"2023-07-14T18:23:10.610671+0000","last_fullsized":"2023-07-14T18:23:10.610671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:08.072924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227886+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:23:11.227886+0000","last_peered":"2023-07-14T18:23:11.227886+0000","last_clean":"2023-07-14T18:23:11.227886+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:23:11.227886+0000","last_undegraded":"2023-07-14T18:23:11.227886+0000","last_fullsized":"2023-07-14T18:23:11.227886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:15.483937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444815+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:23:10.444815+0000","last_peered":"2023-07-14T18:23:10.444815+0000","last_clean":"2023-07-14T18:23:10.444815+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:23:10.444815+0000","last_undegraded":"2023-07-14T18:23:10.444815+0000","last_fullsized":"2023-07-14T18:23:10.444815+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:06.195971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226320+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:23:11.226320+0000","last_peered":"2023-07-14T18:23:11.226320+0000","last_clean":"2023-07-14T18:23:11.226320+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:23:11.226320+0000","last_undegraded":"2023-07-14T18:23:11.226320+0000","last_fullsized":"2023-07-14T18:23:11.226320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:40.019875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226467+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:23:11.226467+0000","last_peered":"2023-07-14T18:23:11.226467+0000","last_clean":"2023-07-14T18:23:11.226467+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:23:11.226467+0000","last_undegraded":"2023-07-14T18:23:11.226467+0000","last_fullsized":"2023-07-14T18:23:11.226467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:46.851772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447060+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:23:10.447060+0000","last_peered":"2023-07-14T18:23:10.447060+0000","last_clean":"2023-07-14T18:23:10.447060+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:23:10.447060+0000","last_undegraded":"2023-07-14T18:23:10.447060+0000","last_fullsized":"2023-07-14T18:23:10.447060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:57.018695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227034+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:23:11.227034+0000","last_peered":"2023-07-14T18:23:11.227034+0000","last_clean":"2023-07-14T18:23:11.227034+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:23:11.227034+0000","last_undegraded":"2023-07-14T18:23:11.227034+0000","last_fullsized":"2023-07-14T18:23:11.227034+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:04.302893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444988+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:23:10.444988+0000","last_peered":"2023-07-14T18:23:10.444988+0000","last_clean":"2023-07-14T18:23:10.444988+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:23:10.444988+0000","last_undegraded":"2023-07-14T18:23:10.444988+0000","last_fullsized":"2023-07-14T18:23:10.444988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:07.815568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445549+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:23:10.445549+0000","last_peered":"2023-07-14T18:23:10.445549+0000","last_clean":"2023-07-14T18:23:10.445549+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:23:10.445549+0000","last_undegraded":"2023-07-14T18:23:10.445549+0000","last_fullsized":"2023-07-14T18:23:10.445549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:26.709513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445503+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:23:10.445503+0000","last_peered":"2023-07-14T18:23:10.445503+0000","last_clean":"2023-07-14T18:23:10.445503+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:23:10.445503+0000","last_undegraded":"2023-07-14T18:23:10.445503+0000","last_fullsized":"2023-07-14T18:23:10.445503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:03.591411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270445+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:23:27.270445+0000","last_peered":"2023-07-14T18:23:27.270445+0000","last_clean":"2023-07-14T18:23:27.270445+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:23:27.270445+0000","last_undegraded":"2023-07-14T18:23:27.270445+0000","last_fullsized":"2023-07-14T18:23:27.270445+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:56.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227501+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:23:11.227501+0000","last_peered":"2023-07-14T18:23:11.227501+0000","last_clean":"2023-07-14T18:23:11.227501+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:23:11.227501+0000","last_undegraded":"2023-07-14T18:23:11.227501+0000","last_fullsized":"2023-07-14T18:23:11.227501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:10.900906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226877+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:23:11.226877+0000","last_peered":"2023-07-14T18:23:11.226877+0000","last_clean":"2023-07-14T18:23:11.226877+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:23:11.226877+0000","last_undegraded":"2023-07-14T18:23:11.226877+0000","last_fullsized":"2023-07-14T18:23:11.226877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:59.578019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227969+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:23:11.227969+0000","last_peered":"2023-07-14T18:23:11.227969+0000","last_clean":"2023-07-14T18:23:11.227969+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:23:11.227969+0000","last_undegraded":"2023-07-14T18:23:11.227969+0000","last_fullsized":"2023-07-14T18:23:11.227969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:23.742307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226855+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:23:11.226855+0000","last_peered":"2023-07-14T18:23:11.226855+0000","last_clean":"2023-07-14T18:23:11.226855+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:23:11.226855+0000","last_undegraded":"2023-07-14T18:23:11.226855+0000","last_fullsized":"2023-07-14T18:23:11.226855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:13.892292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227338+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:23:11.227338+0000","last_peered":"2023-07-14T18:23:11.227338+0000","last_clean":"2023-07-14T18:23:11.227338+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:23:11.227338+0000","last_undegraded":"2023-07-14T18:23:11.227338+0000","last_fullsized":"2023-07-14T18:23:11.227338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:36:55.710853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263269+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:23:26.263269+0000","last_peered":"2023-07-14T18:23:26.263269+0000","last_clean":"2023-07-14T18:23:26.263269+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:23:26.263269+0000","last_undegraded":"2023-07-14T18:23:26.263269+0000","last_fullsized":"2023-07-14T18:23:26.263269+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:30.855601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447648+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:23:10.447648+0000","last_peered":"2023-07-14T18:23:10.447648+0000","last_clean":"2023-07-14T18:23:10.447648+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:23:10.447648+0000","last_undegraded":"2023-07-14T18:23:10.447648+0000","last_fullsized":"2023-07-14T18:23:10.447648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:35.121798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227220+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:23:11.227220+0000","last_peered":"2023-07-14T18:23:11.227220+0000","last_clean":"2023-07-14T18:23:11.227220+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:23:11.227220+0000","last_undegraded":"2023-07-14T18:23:11.227220+0000","last_fullsized":"2023-07-14T18:23:11.227220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:10.747246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226561+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:23:11.226561+0000","last_peered":"2023-07-14T18:23:11.226561+0000","last_clean":"2023-07-14T18:23:11.226561+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:23:11.226561+0000","last_undegraded":"2023-07-14T18:23:11.226561+0000","last_fullsized":"2023-07-14T18:23:11.226561+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:38.727870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226921+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:23:11.226921+0000","last_peered":"2023-07-14T18:23:11.226921+0000","last_clean":"2023-07-14T18:23:11.226921+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:23:11.226921+0000","last_undegraded":"2023-07-14T18:23:11.226921+0000","last_fullsized":"2023-07-14T18:23:11.226921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:28:05.734032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445429+0000","last_change":"2023-07-14T18:23:09.448538+0000","last_active":"2023-07-14T18:23:10.445429+0000","last_peered":"2023-07-14T18:23:10.445429+0000","last_clean":"2023-07-14T18:23:10.445429+0000","last_became_active":"2023-07-14T18:23:09.448108+0000","last_became_peered":"2023-07-14T18:23:09.448108+0000","last_unstale":"2023-07-14T18:23:10.445429+0000","last_undegraded":"2023-07-14T18:23:10.445429+0000","last_fullsized":"2023-07-14T18:23:10.445429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:11.225500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.244900+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:23:24.244900+0000","last_peered":"2023-07-14T18:23:24.244900+0000","last_clean":"2023-07-14T18:23:24.244900+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:23:24.244900+0000","last_undegraded":"2023-07-14T18:23:24.244900+0000","last_fullsized":"2023-07-14T18:23:24.244900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:13.492386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226906+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:23:11.226906+0000","last_peered":"2023-07-14T18:23:11.226906+0000","last_clean":"2023-07-14T18:23:11.226906+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:23:11.226906+0000","last_undegraded":"2023-07-14T18:23:11.226906+0000","last_fullsized":"2023-07-14T18:23:11.226906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:51:56.191875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445464+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:23:10.445464+0000","last_peered":"2023-07-14T18:23:10.445464+0000","last_clean":"2023-07-14T18:23:10.445464+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:23:10.445464+0000","last_undegraded":"2023-07-14T18:23:10.445464+0000","last_fullsized":"2023-07-14T18:23:10.445464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:14.943827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263216+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:23:26.263216+0000","last_peered":"2023-07-14T18:23:26.263216+0000","last_clean":"2023-07-14T18:23:26.263216+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:23:26.263216+0000","last_undegraded":"2023-07-14T18:23:26.263216+0000","last_fullsized":"2023-07-14T18:23:26.263216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:41.156493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229469+0000","last_change":"2023-07-14T18:23:05.254706+0000","last_active":"2023-07-14T18:23:11.229469+0000","last_peered":"2023-07-14T18:23:11.229469+0000","last_clean":"2023-07-14T18:23:11.229469+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.229469+0000","last_undegraded":"2023-07-14T18:23:11.229469+0000","last_fullsized":"2023-07-14T18:23:11.229469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:06:59.136778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227564+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:23:11.227564+0000","last_peered":"2023-07-14T18:23:11.227564+0000","last_clean":"2023-07-14T18:23:11.227564+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:23:11.227564+0000","last_undegraded":"2023-07-14T18:23:11.227564+0000","last_fullsized":"2023-07-14T18:23:11.227564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:44.645790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.254386+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:23:25.254386+0000","last_peered":"2023-07-14T18:23:25.254386+0000","last_clean":"2023-07-14T18:23:25.254386+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:23:25.254386+0000","last_undegraded":"2023-07-14T18:23:25.254386+0000","last_fullsized":"2023-07-14T18:23:25.254386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:35.962587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226991+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:23:11.226991+0000","last_peered":"2023-07-14T18:23:11.226991+0000","last_clean":"2023-07-14T18:23:11.226991+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:23:11.226991+0000","last_undegraded":"2023-07-14T18:23:11.226991+0000","last_fullsized":"2023-07-14T18:23:11.226991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:17.102026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226340+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:23:11.226340+0000","last_peered":"2023-07-14T18:23:11.226340+0000","last_clean":"2023-07-14T18:23:11.226340+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:23:11.226340+0000","last_undegraded":"2023-07-14T18:23:11.226340+0000","last_fullsized":"2023-07-14T18:23:11.226340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:06.464493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.262215+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:23:26.262215+0000","last_peered":"2023-07-14T18:23:26.262215+0000","last_clean":"2023-07-14T18:23:26.262215+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:23:26.262215+0000","last_undegraded":"2023-07-14T18:23:26.262215+0000","last_fullsized":"2023-07-14T18:23:26.262215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:28.512912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444104+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:23:10.444104+0000","last_peered":"2023-07-14T18:23:10.444104+0000","last_clean":"2023-07-14T18:23:10.444104+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:23:10.444104+0000","last_undegraded":"2023-07-14T18:23:10.444104+0000","last_fullsized":"2023-07-14T18:23:10.444104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:14.783284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227144+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:23:11.227144+0000","last_peered":"2023-07-14T18:23:11.227144+0000","last_clean":"2023-07-14T18:23:11.227144+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:23:11.227144+0000","last_undegraded":"2023-07-14T18:23:11.227144+0000","last_fullsized":"2023-07-14T18:23:11.227144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:11:45.259664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.244682+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:23:24.244682+0000","last_peered":"2023-07-14T18:23:24.244682+0000","last_clean":"2023-07-14T18:23:24.244682+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:23:24.244682+0000","last_undegraded":"2023-07-14T18:23:24.244682+0000","last_fullsized":"2023-07-14T18:23:24.244682+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:59.026085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447741+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:23:10.447741+0000","last_peered":"2023-07-14T18:23:10.447741+0000","last_clean":"2023-07-14T18:23:10.447741+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:23:10.447741+0000","last_undegraded":"2023-07-14T18:23:10.447741+0000","last_fullsized":"2023-07-14T18:23:10.447741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:46.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.443969+0000","last_change":"2023-07-14T18:23:09.450239+0000","last_active":"2023-07-14T18:23:10.443969+0000","last_peered":"2023-07-14T18:23:10.443969+0000","last_clean":"2023-07-14T18:23:10.443969+0000","last_became_active":"2023-07-14T18:23:09.449827+0000","last_became_peered":"2023-07-14T18:23:09.449827+0000","last_unstale":"2023-07-14T18:23:10.443969+0000","last_undegraded":"2023-07-14T18:23:10.443969+0000","last_fullsized":"2023-07-14T18:23:10.443969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:45:48.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270511+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:23:27.270511+0000","last_peered":"2023-07-14T18:23:27.270511+0000","last_clean":"2023-07-14T18:23:27.270511+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:23:27.270511+0000","last_undegraded":"2023-07-14T18:23:27.270511+0000","last_fullsized":"2023-07-14T18:23:27.270511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:16.477875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226974+0000","last_change":"2023-07-14T18:23:05.255321+0000","last_active":"2023-07-14T18:23:11.226974+0000","last_peered":"2023-07-14T18:23:11.226974+0000","last_clean":"2023-07-14T18:23:11.226974+0000","last_became_active":"2023-07-14T18:23:05.254443+0000","last_became_peered":"2023-07-14T18:23:05.254443+0000","last_unstale":"2023-07-14T18:23:11.226974+0000","last_undegraded":"2023-07-14T18:23:11.226974+0000","last_fullsized":"2023-07-14T18:23:11.226974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:47:37.938549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227625+0000","last_change":"2023-07-14T18:23:09.448642+0000","last_active":"2023-07-14T18:23:11.227625+0000","last_peered":"2023-07-14T18:23:11.227625+0000","last_clean":"2023-07-14T18:23:11.227625+0000","last_became_active":"2023-07-14T18:23:09.448079+0000","last_became_peered":"2023-07-14T18:23:09.448079+0000","last_unstale":"2023-07-14T18:23:11.227625+0000","last_undegraded":"2023-07-14T18:23:11.227625+0000","last_fullsized":"2023-07-14T18:23:11.227625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:25:39.875566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242725+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:23:24.242725+0000","last_peered":"2023-07-14T18:23:24.242725+0000","last_clean":"2023-07-14T18:23:24.242725+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:23:24.242725+0000","last_undegraded":"2023-07-14T18:23:24.242725+0000","last_fullsized":"2023-07-14T18:23:24.242725+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:35.880799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447624+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:23:10.447624+0000","last_peered":"2023-07-14T18:23:10.447624+0000","last_clean":"2023-07-14T18:23:10.447624+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:23:10.447624+0000","last_undegraded":"2023-07-14T18:23:10.447624+0000","last_fullsized":"2023-07-14T18:23:10.447624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:01.328580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444552+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:23:10.444552+0000","last_peered":"2023-07-14T18:23:10.444552+0000","last_clean":"2023-07-14T18:23:10.444552+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:23:10.444552+0000","last_undegraded":"2023-07-14T18:23:10.444552+0000","last_fullsized":"2023-07-14T18:23:10.444552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:07:35.036816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'12","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.243600+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:23:24.243600+0000","last_peered":"2023-07-14T18:23:24.243600+0000","last_clean":"2023-07-14T18:23:24.243600+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:23:24.243600+0000","last_undegraded":"2023-07-14T18:23:24.243600+0000","last_fullsized":"2023-07-14T18:23:24.243600+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:45.329055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228668+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:23:11.228668+0000","last_peered":"2023-07-14T18:23:11.228668+0000","last_clean":"2023-07-14T18:23:11.228668+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:23:11.228668+0000","last_undegraded":"2023-07-14T18:23:11.228668+0000","last_fullsized":"2023-07-14T18:23:11.228668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:23.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227342+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:23:11.227342+0000","last_peered":"2023-07-14T18:23:11.227342+0000","last_clean":"2023-07-14T18:23:11.227342+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:23:11.227342+0000","last_undegraded":"2023-07-14T18:23:11.227342+0000","last_fullsized":"2023-07-14T18:23:11.227342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:33:09.226720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.269944+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:23:27.269944+0000","last_peered":"2023-07-14T18:23:27.269944+0000","last_clean":"2023-07-14T18:23:27.269944+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:23:27.269944+0000","last_undegraded":"2023-07-14T18:23:27.269944+0000","last_fullsized":"2023-07-14T18:23:27.269944+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:15.877531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444561+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:23:10.444561+0000","last_peered":"2023-07-14T18:23:10.444561+0000","last_clean":"2023-07-14T18:23:10.444561+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:23:10.444561+0000","last_undegraded":"2023-07-14T18:23:10.444561+0000","last_fullsized":"2023-07-14T18:23:10.444561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:11:50.717670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226859+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:23:11.226859+0000","last_peered":"2023-07-14T18:23:11.226859+0000","last_clean":"2023-07-14T18:23:11.226859+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:23:11.226859+0000","last_undegraded":"2023-07-14T18:23:11.226859+0000","last_fullsized":"2023-07-14T18:23:11.226859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:40:53.808997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263214+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:23:26.263214+0000","last_peered":"2023-07-14T18:23:26.263214+0000","last_clean":"2023-07-14T18:23:26.263214+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:23:26.263214+0000","last_undegraded":"2023-07-14T18:23:26.263214+0000","last_fullsized":"2023-07-14T18:23:26.263214+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:25:36.450416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610444+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:23:10.610444+0000","last_peered":"2023-07-14T18:23:10.610444+0000","last_clean":"2023-07-14T18:23:10.610444+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:23:10.610444+0000","last_undegraded":"2023-07-14T18:23:10.610444+0000","last_fullsized":"2023-07-14T18:23:10.610444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:21.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444636+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:23:10.444636+0000","last_peered":"2023-07-14T18:23:10.444636+0000","last_clean":"2023-07-14T18:23:10.444636+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:23:10.444636+0000","last_undegraded":"2023-07-14T18:23:10.444636+0000","last_fullsized":"2023-07-14T18:23:10.444636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:41.687245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227015+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:23:11.227015+0000","last_peered":"2023-07-14T18:23:11.227015+0000","last_clean":"2023-07-14T18:23:11.227015+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:23:11.227015+0000","last_undegraded":"2023-07-14T18:23:11.227015+0000","last_fullsized":"2023-07-14T18:23:11.227015+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:54:03.971193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610150+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:23:10.610150+0000","last_peered":"2023-07-14T18:23:10.610150+0000","last_clean":"2023-07-14T18:23:10.610150+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:23:10.610150+0000","last_undegraded":"2023-07-14T18:23:10.610150+0000","last_fullsized":"2023-07-14T18:23:10.610150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:26.204125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445599+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:23:10.445599+0000","last_peered":"2023-07-14T18:23:10.445599+0000","last_clean":"2023-07-14T18:23:10.445599+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:23:10.445599+0000","last_undegraded":"2023-07-14T18:23:10.445599+0000","last_fullsized":"2023-07-14T18:23:10.445599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:32:00.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'16","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.252676+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:23:25.252676+0000","last_peered":"2023-07-14T18:23:25.252676+0000","last_clean":"2023-07-14T18:23:25.252676+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:23:25.252676+0000","last_undegraded":"2023-07-14T18:23:25.252676+0000","last_fullsized":"2023-07-14T18:23:25.252676+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:45.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228441+0000","last_change":"2023-07-14T18:23:05.255683+0000","last_active":"2023-07-14T18:23:11.228441+0000","last_peered":"2023-07-14T18:23:11.228441+0000","last_clean":"2023-07-14T18:23:11.228441+0000","last_became_active":"2023-07-14T18:23:05.255360+0000","last_became_peered":"2023-07-14T18:23:05.255360+0000","last_unstale":"2023-07-14T18:23:11.228441+0000","last_undegraded":"2023-07-14T18:23:11.228441+0000","last_fullsized":"2023-07-14T18:23:11.228441+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:23.880165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228337+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:23:11.228337+0000","last_peered":"2023-07-14T18:23:11.228337+0000","last_clean":"2023-07-14T18:23:11.228337+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:23:11.228337+0000","last_undegraded":"2023-07-14T18:23:11.228337+0000","last_fullsized":"2023-07-14T18:23:11.228337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:18.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226684+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:23:11.226684+0000","last_peered":"2023-07-14T18:23:11.226684+0000","last_clean":"2023-07-14T18:23:11.226684+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:23:11.226684+0000","last_undegraded":"2023-07-14T18:23:11.226684+0000","last_fullsized":"2023-07-14T18:23:11.226684+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:33.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227454+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:23:11.227454+0000","last_peered":"2023-07-14T18:23:11.227454+0000","last_clean":"2023-07-14T18:23:11.227454+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.227454+0000","last_undegraded":"2023-07-14T18:23:11.227454+0000","last_fullsized":"2023-07-14T18:23:11.227454+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:55.074433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226809+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:23:11.226809+0000","last_peered":"2023-07-14T18:23:11.226809+0000","last_clean":"2023-07-14T18:23:11.226809+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:23:11.226809+0000","last_undegraded":"2023-07-14T18:23:11.226809+0000","last_fullsized":"2023-07-14T18:23:11.226809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:44.909597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444437+0000","last_change":"2023-07-14T18:23:04.418035+0000","last_active":"2023-07-14T18:23:10.444437+0000","last_peered":"2023-07-14T18:23:10.444437+0000","last_clean":"2023-07-14T18:23:10.444437+0000","last_became_active":"2023-07-14T18:23:04.417645+0000","last_became_peered":"2023-07-14T18:23:04.417645+0000","last_unstale":"2023-07-14T18:23:10.444437+0000","last_undegraded":"2023-07-14T18:23:10.444437+0000","last_fullsized":"2023-07-14T18:23:10.444437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:08.545074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227920+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:23:11.227920+0000","last_peered":"2023-07-14T18:23:11.227920+0000","last_clean":"2023-07-14T18:23:11.227920+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:23:11.227920+0000","last_undegraded":"2023-07-14T18:23:11.227920+0000","last_fullsized":"2023-07-14T18:23:11.227920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:47.382928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445312+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:23:10.445312+0000","last_peered":"2023-07-14T18:23:10.445312+0000","last_clean":"2023-07-14T18:23:10.445312+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:23:10.445312+0000","last_undegraded":"2023-07-14T18:23:10.445312+0000","last_fullsized":"2023-07-14T18:23:10.445312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:36:47.920949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.290262+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:23:28.290262+0000","last_peered":"2023-07-14T18:23:28.290262+0000","last_clean":"2023-07-14T18:23:28.290262+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:23:28.290262+0000","last_undegraded":"2023-07-14T18:23:28.290262+0000","last_fullsized":"2023-07-14T18:23:28.290262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:22:04.367333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227207+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:23:11.227207+0000","last_peered":"2023-07-14T18:23:11.227207+0000","last_clean":"2023-07-14T18:23:11.227207+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:23:11.227207+0000","last_undegraded":"2023-07-14T18:23:11.227207+0000","last_fullsized":"2023-07-14T18:23:11.227207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:56:42.916396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445652+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:23:10.445652+0000","last_peered":"2023-07-14T18:23:10.445652+0000","last_clean":"2023-07-14T18:23:10.445652+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:23:10.445652+0000","last_undegraded":"2023-07-14T18:23:10.445652+0000","last_fullsized":"2023-07-14T18:23:10.445652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:51:24.527443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228927+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:23:11.228927+0000","last_peered":"2023-07-14T18:23:11.228927+0000","last_clean":"2023-07-14T18:23:11.228927+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:23:11.228927+0000","last_undegraded":"2023-07-14T18:23:11.228927+0000","last_fullsized":"2023-07-14T18:23:11.228927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:15.545609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.287546+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:23:28.287546+0000","last_peered":"2023-07-14T18:23:28.287546+0000","last_clean":"2023-07-14T18:23:28.287546+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:23:28.287546+0000","last_undegraded":"2023-07-14T18:23:28.287546+0000","last_fullsized":"2023-07-14T18:23:28.287546+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:33:45.754899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610640+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:23:10.610640+0000","last_peered":"2023-07-14T18:23:10.610640+0000","last_clean":"2023-07-14T18:23:10.610640+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:23:10.610640+0000","last_undegraded":"2023-07-14T18:23:10.610640+0000","last_fullsized":"2023-07-14T18:23:10.610640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:17.385324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226985+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:23:11.226985+0000","last_peered":"2023-07-14T18:23:11.226985+0000","last_clean":"2023-07-14T18:23:11.226985+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:23:11.226985+0000","last_undegraded":"2023-07-14T18:23:11.226985+0000","last_fullsized":"2023-07-14T18:23:11.226985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:48.279521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242903+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:23:24.242903+0000","last_peered":"2023-07-14T18:23:24.242903+0000","last_clean":"2023-07-14T18:23:24.242903+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:23:24.242903+0000","last_undegraded":"2023-07-14T18:23:24.242903+0000","last_fullsized":"2023-07-14T18:23:24.242903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:52.919560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.609987+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:23:10.609987+0000","last_peered":"2023-07-14T18:23:10.609987+0000","last_clean":"2023-07-14T18:23:10.609987+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:23:10.609987+0000","last_undegraded":"2023-07-14T18:23:10.609987+0000","last_fullsized":"2023-07-14T18:23:10.609987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:43.598474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227359+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:23:11.227359+0000","last_peered":"2023-07-14T18:23:11.227359+0000","last_clean":"2023-07-14T18:23:11.227359+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:23:11.227359+0000","last_undegraded":"2023-07-14T18:23:11.227359+0000","last_fullsized":"2023-07-14T18:23:11.227359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:34:29.533657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444959+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:23:10.444959+0000","last_peered":"2023-07-14T18:23:10.444959+0000","last_clean":"2023-07-14T18:23:10.444959+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:23:10.444959+0000","last_undegraded":"2023-07-14T18:23:10.444959+0000","last_fullsized":"2023-07-14T18:23:10.444959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:08.456204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.252048+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:23:25.252048+0000","last_peered":"2023-07-14T18:23:25.252048+0000","last_clean":"2023-07-14T18:23:25.252048+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:23:25.252048+0000","last_undegraded":"2023-07-14T18:23:25.252048+0000","last_fullsized":"2023-07-14T18:23:25.252048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:57.271343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226827+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:23:11.226827+0000","last_peered":"2023-07-14T18:23:11.226827+0000","last_clean":"2023-07-14T18:23:11.226827+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:23:11.226827+0000","last_undegraded":"2023-07-14T18:23:11.226827+0000","last_fullsized":"2023-07-14T18:23:11.226827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:41.538513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'3","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263857+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:23:26.263857+0000","last_peered":"2023-07-14T18:23:26.263857+0000","last_clean":"2023-07-14T18:23:26.263857+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:23:26.263857+0000","last_undegraded":"2023-07-14T18:23:26.263857+0000","last_fullsized":"2023-07-14T18:23:26.263857+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:29.140961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445507+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:23:10.445507+0000","last_peered":"2023-07-14T18:23:10.445507+0000","last_clean":"2023-07-14T18:23:10.445507+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:23:10.445507+0000","last_undegraded":"2023-07-14T18:23:10.445507+0000","last_fullsized":"2023-07-14T18:23:10.445507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:01.714171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226933+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:23:11.226933+0000","last_peered":"2023-07-14T18:23:11.226933+0000","last_clean":"2023-07-14T18:23:11.226933+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:23:11.226933+0000","last_undegraded":"2023-07-14T18:23:11.226933+0000","last_fullsized":"2023-07-14T18:23:11.226933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:06:06.174345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263560+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:23:26.263560+0000","last_peered":"2023-07-14T18:23:26.263560+0000","last_clean":"2023-07-14T18:23:26.263560+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:23:26.263560+0000","last_undegraded":"2023-07-14T18:23:26.263560+0000","last_fullsized":"2023-07-14T18:23:26.263560+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:38.063154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227450+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:23:11.227450+0000","last_peered":"2023-07-14T18:23:11.227450+0000","last_clean":"2023-07-14T18:23:11.227450+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:23:11.227450+0000","last_undegraded":"2023-07-14T18:23:11.227450+0000","last_fullsized":"2023-07-14T18:23:11.227450+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:39.395014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445040+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:23:10.445040+0000","last_peered":"2023-07-14T18:23:10.445040+0000","last_clean":"2023-07-14T18:23:10.445040+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:23:10.445040+0000","last_undegraded":"2023-07-14T18:23:10.445040+0000","last_fullsized":"2023-07-14T18:23:10.445040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:54.870365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.243023+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:23:24.243023+0000","last_peered":"2023-07-14T18:23:24.243023+0000","last_clean":"2023-07-14T18:23:24.243023+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:23:24.243023+0000","last_undegraded":"2023-07-14T18:23:24.243023+0000","last_fullsized":"2023-07-14T18:23:24.243023+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:54:38.614796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226909+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:23:11.226909+0000","last_peered":"2023-07-14T18:23:11.226909+0000","last_clean":"2023-07-14T18:23:11.226909+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:23:11.226909+0000","last_undegraded":"2023-07-14T18:23:11.226909+0000","last_fullsized":"2023-07-14T18:23:11.226909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:43.634763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227411+0000","last_change":"2023-07-14T18:23:09.449832+0000","last_active":"2023-07-14T18:23:11.227411+0000","last_peered":"2023-07-14T18:23:11.227411+0000","last_clean":"2023-07-14T18:23:11.227411+0000","last_became_active":"2023-07-14T18:23:09.448997+0000","last_became_peered":"2023-07-14T18:23:09.448997+0000","last_unstale":"2023-07-14T18:23:11.227411+0000","last_undegraded":"2023-07-14T18:23:11.227411+0000","last_fullsized":"2023-07-14T18:23:11.227411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:56.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242161+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:23:24.242161+0000","last_peered":"2023-07-14T18:23:24.242161+0000","last_clean":"2023-07-14T18:23:24.242161+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:23:24.242161+0000","last_undegraded":"2023-07-14T18:23:24.242161+0000","last_fullsized":"2023-07-14T18:23:24.242161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:46.321986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226403+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:23:11.226403+0000","last_peered":"2023-07-14T18:23:11.226403+0000","last_clean":"2023-07-14T18:23:11.226403+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:23:11.226403+0000","last_undegraded":"2023-07-14T18:23:11.226403+0000","last_fullsized":"2023-07-14T18:23:11.226403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:03:27.653641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444903+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:23:10.444903+0000","last_peered":"2023-07-14T18:23:10.444903+0000","last_clean":"2023-07-14T18:23:10.444903+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:23:10.444903+0000","last_undegraded":"2023-07-14T18:23:10.444903+0000","last_fullsized":"2023-07-14T18:23:10.444903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:40.273437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268661+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:23:27.268661+0000","last_peered":"2023-07-14T18:23:27.268661+0000","last_clean":"2023-07-14T18:23:27.268661+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:23:27.268661+0000","last_undegraded":"2023-07-14T18:23:27.268661+0000","last_fullsized":"2023-07-14T18:23:27.268661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:25.573588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229498+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:23:11.229498+0000","last_peered":"2023-07-14T18:23:11.229498+0000","last_clean":"2023-07-14T18:23:11.229498+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:23:11.229498+0000","last_undegraded":"2023-07-14T18:23:11.229498+0000","last_fullsized":"2023-07-14T18:23:11.229498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:55.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227675+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:23:11.227675+0000","last_peered":"2023-07-14T18:23:11.227675+0000","last_clean":"2023-07-14T18:23:11.227675+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:23:11.227675+0000","last_undegraded":"2023-07-14T18:23:11.227675+0000","last_fullsized":"2023-07-14T18:23:11.227675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:26:45.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445140+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:23:10.445140+0000","last_peered":"2023-07-14T18:23:10.445140+0000","last_clean":"2023-07-14T18:23:10.445140+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:23:10.445140+0000","last_undegraded":"2023-07-14T18:23:10.445140+0000","last_fullsized":"2023-07-14T18:23:10.445140+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:57:27.449607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227548+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:23:11.227548+0000","last_peered":"2023-07-14T18:23:11.227548+0000","last_clean":"2023-07-14T18:23:11.227548+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:23:11.227548+0000","last_undegraded":"2023-07-14T18:23:11.227548+0000","last_fullsized":"2023-07-14T18:23:11.227548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:58.508966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227371+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:23:11.227371+0000","last_peered":"2023-07-14T18:23:11.227371+0000","last_clean":"2023-07-14T18:23:11.227371+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:23:11.227371+0000","last_undegraded":"2023-07-14T18:23:11.227371+0000","last_fullsized":"2023-07-14T18:23:11.227371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:20.402932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.241780+0000","last_change":"2023-07-14T18:23:04.424951+0000","last_active":"2023-07-14T18:23:24.241780+0000","last_peered":"2023-07-14T18:23:24.241780+0000","last_clean":"2023-07-14T18:23:24.241780+0000","last_became_active":"2023-07-14T18:23:04.424641+0000","last_became_peered":"2023-07-14T18:23:04.424641+0000","last_unstale":"2023-07-14T18:23:24.241780+0000","last_undegraded":"2023-07-14T18:23:24.241780+0000","last_fullsized":"2023-07-14T18:23:24.241780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:25.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610293+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:23:10.610293+0000","last_peered":"2023-07-14T18:23:10.610293+0000","last_clean":"2023-07-14T18:23:10.610293+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:23:10.610293+0000","last_undegraded":"2023-07-14T18:23:10.610293+0000","last_fullsized":"2023-07-14T18:23:10.610293+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:11.675787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447265+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:23:10.447265+0000","last_peered":"2023-07-14T18:23:10.447265+0000","last_clean":"2023-07-14T18:23:10.447265+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:23:10.447265+0000","last_undegraded":"2023-07-14T18:23:10.447265+0000","last_fullsized":"2023-07-14T18:23:10.447265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:43.737345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.265288+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:23:26.265288+0000","last_peered":"2023-07-14T18:23:26.265288+0000","last_clean":"2023-07-14T18:23:26.265288+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:23:26.265288+0000","last_undegraded":"2023-07-14T18:23:26.265288+0000","last_fullsized":"2023-07-14T18:23:26.265288+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:04.272902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610785+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:23:10.610785+0000","last_peered":"2023-07-14T18:23:10.610785+0000","last_clean":"2023-07-14T18:23:10.610785+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:23:10.610785+0000","last_undegraded":"2023-07-14T18:23:10.610785+0000","last_fullsized":"2023-07-14T18:23:10.610785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:27:06.003284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226097+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:23:11.226097+0000","last_peered":"2023-07-14T18:23:11.226097+0000","last_clean":"2023-07-14T18:23:11.226097+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:23:11.226097+0000","last_undegraded":"2023-07-14T18:23:11.226097+0000","last_fullsized":"2023-07-14T18:23:11.226097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:26.374670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263332+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:23:26.263332+0000","last_peered":"2023-07-14T18:23:26.263332+0000","last_clean":"2023-07-14T18:23:26.263332+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:23:26.263332+0000","last_undegraded":"2023-07-14T18:23:26.263332+0000","last_fullsized":"2023-07-14T18:23:26.263332+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:05:36.820471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610456+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:23:10.610456+0000","last_peered":"2023-07-14T18:23:10.610456+0000","last_clean":"2023-07-14T18:23:10.610456+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:23:10.610456+0000","last_undegraded":"2023-07-14T18:23:10.610456+0000","last_fullsized":"2023-07-14T18:23:10.610456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:55:16.680648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228516+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:23:11.228516+0000","last_peered":"2023-07-14T18:23:11.228516+0000","last_clean":"2023-07-14T18:23:11.228516+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:23:11.228516+0000","last_undegraded":"2023-07-14T18:23:11.228516+0000","last_fullsized":"2023-07-14T18:23:11.228516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:45.616005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268218+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:23:27.268218+0000","last_peered":"2023-07-14T18:23:27.268218+0000","last_clean":"2023-07-14T18:23:27.268218+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:23:27.268218+0000","last_undegraded":"2023-07-14T18:23:27.268218+0000","last_fullsized":"2023-07-14T18:23:27.268218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:35.670294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.611027+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:23:10.611027+0000","last_peered":"2023-07-14T18:23:10.611027+0000","last_clean":"2023-07-14T18:23:10.611027+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:23:10.611027+0000","last_undegraded":"2023-07-14T18:23:10.611027+0000","last_fullsized":"2023-07-14T18:23:10.611027+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:51.422332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229684+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:23:11.229684+0000","last_peered":"2023-07-14T18:23:11.229684+0000","last_clean":"2023-07-14T18:23:11.229684+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:23:11.229684+0000","last_undegraded":"2023-07-14T18:23:11.229684+0000","last_fullsized":"2023-07-14T18:23:11.229684+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:17.955330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445263+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:23:10.445263+0000","last_peered":"2023-07-14T18:23:10.445263+0000","last_clean":"2023-07-14T18:23:10.445263+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:23:10.445263+0000","last_undegraded":"2023-07-14T18:23:10.445263+0000","last_fullsized":"2023-07-14T18:23:10.445263+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:47.687598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445138+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:23:10.445138+0000","last_peered":"2023-07-14T18:23:10.445138+0000","last_clean":"2023-07-14T18:23:10.445138+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:23:10.445138+0000","last_undegraded":"2023-07-14T18:23:10.445138+0000","last_fullsized":"2023-07-14T18:23:10.445138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:22.609183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226339+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:23:11.226339+0000","last_peered":"2023-07-14T18:23:11.226339+0000","last_clean":"2023-07-14T18:23:11.226339+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:23:11.226339+0000","last_undegraded":"2023-07-14T18:23:11.226339+0000","last_fullsized":"2023-07-14T18:23:11.226339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:45.583600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445397+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:23:10.445397+0000","last_peered":"2023-07-14T18:23:10.445397+0000","last_clean":"2023-07-14T18:23:10.445397+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:23:10.445397+0000","last_undegraded":"2023-07-14T18:23:10.445397+0000","last_fullsized":"2023-07-14T18:23:10.445397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:10.666041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227616+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:23:11.227616+0000","last_peered":"2023-07-14T18:23:11.227616+0000","last_clean":"2023-07-14T18:23:11.227616+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:23:11.227616+0000","last_undegraded":"2023-07-14T18:23:11.227616+0000","last_fullsized":"2023-07-14T18:23:11.227616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:50.867762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445702+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:23:10.445702+0000","last_peered":"2023-07-14T18:23:10.445702+0000","last_clean":"2023-07-14T18:23:10.445702+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:23:10.445702+0000","last_undegraded":"2023-07-14T18:23:10.445702+0000","last_fullsized":"2023-07-14T18:23:10.445702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:50.705252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227634+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:23:11.227634+0000","last_peered":"2023-07-14T18:23:11.227634+0000","last_clean":"2023-07-14T18:23:11.227634+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:23:11.227634+0000","last_undegraded":"2023-07-14T18:23:11.227634+0000","last_fullsized":"2023-07-14T18:23:11.227634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:57.010106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228805+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:23:11.228805+0000","last_peered":"2023-07-14T18:23:11.228805+0000","last_clean":"2023-07-14T18:23:11.228805+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:23:11.228805+0000","last_undegraded":"2023-07-14T18:23:11.228805+0000","last_fullsized":"2023-07-14T18:23:11.228805+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:27.226698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227229+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:23:11.227229+0000","last_peered":"2023-07-14T18:23:11.227229+0000","last_clean":"2023-07-14T18:23:11.227229+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:23:11.227229+0000","last_undegraded":"2023-07-14T18:23:11.227229+0000","last_fullsized":"2023-07-14T18:23:11.227229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:50.246761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447443+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:23:10.447443+0000","last_peered":"2023-07-14T18:23:10.447443+0000","last_clean":"2023-07-14T18:23:10.447443+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:23:10.447443+0000","last_undegraded":"2023-07-14T18:23:10.447443+0000","last_fullsized":"2023-07-14T18:23:10.447443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:22.891172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444900+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:23:10.444900+0000","last_peered":"2023-07-14T18:23:10.444900+0000","last_clean":"2023-07-14T18:23:10.444900+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:23:10.444900+0000","last_undegraded":"2023-07-14T18:23:10.444900+0000","last_fullsized":"2023-07-14T18:23:10.444900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:52.929194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227023+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:23:11.227023+0000","last_peered":"2023-07-14T18:23:11.227023+0000","last_clean":"2023-07-14T18:23:11.227023+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:23:11.227023+0000","last_undegraded":"2023-07-14T18:23:11.227023+0000","last_fullsized":"2023-07-14T18:23:11.227023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:19.793596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":20480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":9502,"num_objects":80,"num_object_clones":0,"num_object_copies":240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":179,"num_write_kb":63,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":45621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514511,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":458554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.497},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.745},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.63100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.53900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66200000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67600000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.78400000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.77100000000000002}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.879}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.84099999999999997}]}]},{"osd":10,"up_from":69,"seq":296352743443,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28556,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716596,"statfs":{"total":95995035648,"available":95965794304,"internally_reserved":0,"allocated":1576960,"data_stored":1048740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.73099999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.872}]},{"osd":6,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.92800000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.93799999999999994}]},{"osd":9,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.83599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]}]},{"osd":9,"up_from":64,"seq":274877906968,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":458142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.41199999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.65700000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.499}]},{"osd":3,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.38700000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"last":0.625}]},{"osd":5,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.3799999999999999,"5min":1.3799999999999999,"15min":1.3799999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.2110000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.48299999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.2390000000000001,"5min":1.2390000000000001,"15min":1.2390000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.2889999999999999,"5min":1.2889999999999999,"15min":1.2889999999999999},"last":0.64500000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"last":0.71399999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.747}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135900,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":453662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.56699999999999995}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.433}]},{"osd":3,"last update":"Fri Jul 14 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.82599999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.62},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.65000000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.66800000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.745},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.754}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.88400000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.90600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1054599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.55100000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40500000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.748},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.751,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.502,"15min":0.497},"min":{"1min":0.313,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.59599999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.81799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.51100000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":1.02,"15min":1.02},"last":0.81699999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.99099999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.97099999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82899999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":453325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26100000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.56499999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.40799999999999997,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.41999999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.57499999999999996,"15min":0.56499999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80800000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.75,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.46800000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.62,"15min":0.60999999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.94399999999999995,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.748}]},{"osd":3,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.46500000000000002,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.441},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.505,"15min":0.505},"min":{"1min":0.317,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.39500000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91600000000000004,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.63900000000000001,"15min":0.63300000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86799999999999999,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.68100000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.85999999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.48099999999999998,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.94599999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.61699999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.65200000000000002,"15min":0.64400000000000002},"min":{"1min":0.505,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.68600000000000005,"15min":0.67600000000000005},"min":{"1min":0.50900000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.72299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.75800000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.63100000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.81799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":456718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:55 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.505,"15min":0.501},"min":{"1min":0.39100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.51300000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.86899999999999999,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.53100000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.82799999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.54800000000000004,"15min":0.52700000000000002},"min":{"1min":0.434,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.57199999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.38,"15min":0.38},"max":{"1min":0.78600000000000003,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.55500000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:22:56 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.443,"15min":0.42299999999999999},"min":{"1min":0.32400000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.45500000000000002},"min":{"1min":0.30299999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.64000000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.71499999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.61599999999999999,"15min":0.60499999999999998},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.61899999999999999,"15min":0.60799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.69999999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.58999999999999997,"15min":0.56799999999999995},"min":{"1min":0.42099999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.59299999999999997,"15min":0.57199999999999995},"min":{"1min":0.40100000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.76000000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:22:24 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.60699999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:22:46 2023","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.5,"5min":0.5,"15min":0.5},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.86399999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.84599999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":453345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.54000000000000004,"15min":0.496},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84899999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54800000000000004,"15min":0.51100000000000001},"min":{"1min":0.40799999999999997,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.63}]},{"osd":2,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.89500000000000002,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.502},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56699999999999995,"15min":0.57499999999999996},"min":{"1min":0.36899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88900000000000001,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Jul 14 18:21:34 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.39600000000000002,"15min":0.38200000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.26400000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.439}]},{"osd":4,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.59299999999999997,"15min":0.56299999999999994},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.47599999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.56299999999999994,"15min":0.55300000000000005},"min":{"1min":0.44600000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.46800000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.51700000000000002,"15min":0.50700000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.57399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36599999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52600000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.745}]},{"osd":8,"last update":"Fri Jul 14 18:22:23 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.59899999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.66700000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.85899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411380,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":458594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.375},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:22:38 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.33700000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.54300000000000004,"15min":0.52800000000000002},"min":{"1min":0.374,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:22:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.47799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57299999999999995,"15min":0.55200000000000005},"min":{"1min":0.505,"5min":0.38,"15min":0.38},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.502}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.73399999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60499999999999998,"15min":0.59899999999999998},"min":{"1min":0.46100000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68300000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.753},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.78200000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.76800000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:22:43 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.80600000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.83199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":453073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.37,"15min":0.35999999999999999},"min":{"1min":0.27300000000000002,"5min":0.161,"15min":0.161},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.38100000000000001,"15min":0.35499999999999998},"min":{"1min":0.30599999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.56699999999999995,"15min":0.55700000000000005},"min":{"1min":0.39300000000000002,"5min":0.376,"15min":0.376},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57099999999999995,"15min":0.55900000000000005},"min":{"1min":0.435,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.57799999999999996,"15min":0.55100000000000005},"min":{"1min":0.41399999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.58699999999999997,"15min":0.56399999999999995},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.80900000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:23:00 2023","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.58999999999999997,"15min":0.56499999999999995},"min":{"1min":0.45100000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.59599999999999997,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":0.82599999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.59799999999999998,"15min":0.58699999999999997},"min":{"1min":0.41099999999999998,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.91700000000000004,"5min":1.274,"15min":1.274},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.35399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.58699999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.125,"5min":1.22,"15min":1.22},"last":0.61699999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.042,"5min":1.042,"15min":1.042},"last":0.84199999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.754}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.878}]}]},{"osd":4,"up_from":33,"seq":141733920813,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.40999999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65600000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.62},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.40899999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:40 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.66400000000000003,"15min":0.65500000000000003},"min":{"1min":0.44700000000000001,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.70099999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.89300000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60299999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71799999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.79500000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.75}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757289,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1045151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.437}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.45400000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.65100000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.63100000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:22:41 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.61899999999999999,"15min":0.60499999999999998},"min":{"1min":0.48399999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.64800000000000002,"15min":0.63600000000000001},"min":{"1min":0.53200000000000003,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.77700000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.70399999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:22:44 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.206,"5min":1.206,"15min":1.206},"last":0.72799999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":23831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:23:30.108 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T18:23:30.109 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T18:23:30.109 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T18:23:30.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:30 smithi064 ceph-mon[96720]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:30.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:30 smithi144 ceph-mon[98417]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:30.699 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:30.810 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:23:30 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:23:30.278+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-14T18:23:30.811 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:23:30 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:23:30.278+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T18:23:30.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:30 smithi105 ceph-mon[98013]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:23:31.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:31 smithi064 ceph-mon[96720]: pgmap v267: 129 pgs: 129 active+clean; 586 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 3.9 KiB/s wr, 11 op/s 2023-07-14T18:23:31.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:31 smithi064 ceph-mon[96720]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:31 smithi064 ceph-mon[96720]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:31 smithi144 ceph-mon[98417]: pgmap v267: 129 pgs: 129 active+clean; 586 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 3.9 KiB/s wr, 11 op/s 2023-07-14T18:23:31.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:31 smithi144 ceph-mon[98417]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:31 smithi144 ceph-mon[98417]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:31 smithi105 ceph-mon[98013]: pgmap v267: 129 pgs: 129 active+clean; 586 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s rd, 3.9 KiB/s wr, 11 op/s 2023-07-14T18:23:31.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:31 smithi105 ceph-mon[98013]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:31 smithi105 ceph-mon[98013]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:23:31.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.891+0000 7fc3a6373700 1 -- 172.21.15.64:0/2208910597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 msgr2=0x7fc3a0101330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:31.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.891+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2208910597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0101330 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc394009990 tx=0x7fc394009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 -- 172.21.15.64:0/2208910597 shutdown_connections 2023-07-14T18:23:31.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2208910597 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a0108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2208910597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 0x7fc3a01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2208910597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0101330 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 -- 172.21.15.64:0/2208910597 >> 172.21.15.64:0/2208910597 conn(0x7fc3a00fb330 msgr2=0x7fc3a00fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 -- 172.21.15.64:0/2208910597 shutdown_connections 2023-07-14T18:23:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 -- 172.21.15.64:0/2208910597 wait complete. 2023-07-14T18:23:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 Processor -- start 2023-07-14T18:23:31.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.892+0000 7fc3a6373700 1 -- start start 2023-07-14T18:23:31.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 0x7fc3a0197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:31.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:31.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:31.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a019f680 con 0x7fc3a0100ee0 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a019f7c0 con 0x7fc3a00ffce0 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a6373700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a019f950 con 0x7fc3a0101870 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47486/0 (socket says 172.21.15.64:47486) 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 -- 172.21.15.64:0/2317977165 learned_addr learned my addr 172.21.15.64:0/2317977165 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a4910700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:31.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39ffff700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 0x7fc3a0197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 -- 172.21.15.64:0/2317977165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 msgr2=0x7fc3a019f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a019f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 -- 172.21.15.64:0/2317977165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 msgr2=0x7fc3a0197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 0x7fc3a0197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 -- 172.21.15.64:0/2317977165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3940095f0 con 0x7fc3a0100ee0 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc3a4910700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39f7fe700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc3940057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:31.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39401d070 con 0x7fc3a0100ee0 2023-07-14T18:23:31.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc394022470 con 0x7fc3a0100ee0 2023-07-14T18:23:31.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.893+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39400f6c0 con 0x7fc3a0100ee0 2023-07-14T18:23:31.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.894+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a019fbd0 con 0x7fc3a0100ee0 2023-07-14T18:23:31.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.894+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a019ff90 con 0x7fc3a0100ee0 2023-07-14T18:23:31.951 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.895+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a0073380 con 0x7fc3a0100ee0 2023-07-14T18:23:31.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.896+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc3940225e0 con 0x7fc3a0100ee0 2023-07-14T18:23:31.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.896+0000 7fc39d7fa700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 0x7fc3880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:31.952 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.897+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc3940ce990 con 0x7fc3a0100ee0 2023-07-14T18:23:31.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.897+0000 7fc39ffff700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 0x7fc3880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:31.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.897+0000 7fc39ffff700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 0x7fc3880a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0100d40 tx=0x7fc390009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:31.953 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:31.900+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc394097dc0 con 0x7fc3a0100ee0 2023-07-14T18:23:32.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.175+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc3a01a0640 con 0x7fc3a0100ee0 2023-07-14T18:23:32.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.180+0000 7fc39d7fa700 1 -- 172.21.15.64:0/2317977165 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc39409b3e0 con 0x7fc3a0100ee0 2023-07-14T18:23:32.232 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2023-07-14T18:23:32.232 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQCkkrFkpcyTChAAGHBZUY3NnFCBsqpHJaB7DA== 2023-07-14T18:23:32.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 msgr2=0x7fc3880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 0x7fc3880a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0100d40 tx=0x7fc390009260 comp rx=0 tx=0).stop 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 msgr2=0x7fc3a0198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc394000c00 tx=0x7fc3940057d0 comp rx=0 tx=0).stop 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 shutdown_connections 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc3a0101870 0x7fc3a019f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc3880a11e0 0x7fc3880a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:32.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3a00ffce0 0x7fc3a0197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:32.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 --2- 172.21.15.64:0/2317977165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3a0100ee0 0x7fc3a0198030 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:32.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 >> 172.21.15.64:0/2317977165 conn(0x7fc3a00fb330 msgr2=0x7fc3a00fbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:32.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 shutdown_connections 2023-07-14T18:23:32.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:32.182+0000 7fc3a6373700 1 -- 172.21.15.64:0/2317977165 wait complete. 2023-07-14T18:23:32.564 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:32 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:23:32.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:32 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:23:32.565 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:23:32.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T18:23:32.566 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T18:23:32.607 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T18:23:32.608 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T18:23:32.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T18:23:32.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:32 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:23:32.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:32 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:23:32.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:32 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T18:23:32.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:32 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2317977165' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T18:23:33.205 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:33.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:33 smithi064 ceph-mon[96720]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 15 op/s 2023-07-14T18:23:33.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:33 smithi144 ceph-mon[98417]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 15 op/s 2023-07-14T18:23:33.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:33 smithi105 ceph-mon[98013]: pgmap v268: 129 pgs: 129 active+clean; 587 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4.5 KiB/s wr, 15 op/s 2023-07-14T18:23:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 -- 172.21.15.64:0/1396724156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100030 msgr2=0x7fe180100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 --2- 172.21.15.64:0/1396724156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100030 0x7fe180100400 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe1740099c0 tx=0x7fe174009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 -- 172.21.15.64:0/1396724156 shutdown_connections 2023-07-14T18:23:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 --2- 172.21.15.64:0/1396724156 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1801095e0 0x7fe18010b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 --2- 172.21.15.64:0/1396724156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100030 0x7fe180100400 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 --2- 172.21.15.64:0/1396724156 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe180100940 0x7fe1801090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 -- 172.21.15.64:0/1396724156 >> 172.21.15.64:0/1396724156 conn(0x7fe180075480 msgr2=0x7fe180075880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:34.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 -- 172.21.15.64:0/1396724156 shutdown_connections 2023-07-14T18:23:34.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.376+0000 7fe185653700 1 -- 172.21.15.64:0/1396724156 wait complete. 2023-07-14T18:23:34.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.377+0000 7fe185653700 1 Processor -- start 2023-07-14T18:23:34.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 -- start start 2023-07-14T18:23:34.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe180100030 0x7fe18019aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:34.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:34.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1801095e0 0x7fe180194ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:34.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180195120 con 0x7fe180100940 2023-07-14T18:23:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180195290 con 0x7fe180100030 2023-07-14T18:23:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe185653700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180195450 con 0x7fe1801095e0 2023-07-14T18:23:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:34.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47498/0 (socket says 172.21.15.64:47498) 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 -- 172.21.15.64:0/860615260 learned_addr learned my addr 172.21.15.64:0/860615260 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17f7fe700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1801095e0 0x7fe180194ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17effd700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe180100030 0x7fe18019aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 -- 172.21.15.64:0/860615260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1801095e0 msgr2=0x7fe180194ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1801095e0 0x7fe180194ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 -- 172.21.15.64:0/860615260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe180100030 msgr2=0x7fe18019aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:34.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.378+0000 7fe17e7fc700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe180100030 0x7fe18019aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe17e7fc700 1 -- 172.21.15.64:0/860615260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1740095f0 con 0x7fe180100940 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe17e7fc700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe17000b530 tx=0x7fe17000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170011830 con 0x7fe180100940 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe170011e20 con 0x7fe180100940 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe180195730 con 0x7fe180100940 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe170010dd0 con 0x7fe180100940 2023-07-14T18:23:34.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.379+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1801a1aa0 con 0x7fe180100940 2023-07-14T18:23:34.436 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.380+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18004e340 con 0x7fe180100940 2023-07-14T18:23:34.436 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.381+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe170011990 con 0x7fe180100940 2023-07-14T18:23:34.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.382+0000 7fe167fff700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 0x7fe1680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:34.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.382+0000 7fe17effd700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 0x7fe1680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:34.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.382+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe1700cc750 con 0x7fe180100940 2023-07-14T18:23:34.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.382+0000 7fe17effd700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 0x7fe1680a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe174009f50 tx=0x7fe174005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:34.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.386+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe170095c00 con 0x7fe180100940 2023-07-14T18:23:34.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.638+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fe1801a1cf0 con 0x7fe180100940 2023-07-14T18:23:34.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.643+0000 7fe167fff700 1 -- 172.21.15.64:0/860615260 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260122 (secure 0 0 0) 0x7fe170099220 con 0x7fe180100940 2023-07-14T18:23:34.697 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 msgr2=0x7fe1680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 0x7fe1680a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe174009f50 tx=0x7fe174005fd0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 msgr2=0x7fe18019af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:34.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe17000b530 tx=0x7fe17000b8f0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 shutdown_connections 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe1801095e0 0x7fe180194ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe1680a11e0 0x7fe1680a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe180100030 0x7fe18019aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 --2- 172.21.15.64:0/860615260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe180100940 0x7fe18019af70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 >> 172.21.15.64:0/860615260 conn(0x7fe180075480 msgr2=0x7fe1800fbf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:34.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 shutdown_connections 2023-07-14T18:23:34.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:34.646+0000 7fe185653700 1 -- 172.21.15.64:0/860615260 wait complete. 2023-07-14T18:23:35.263 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":387190897},{"type":"v1","addr":"172.21.15.64:6801","nonce":387190897}]},"active_addr":"172.21.15.64:6801/387190897","active_change":"2023-07-14T18:17:05.380312+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.64:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3470445643}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":2119632363}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3117585401}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1794129224}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":447039006}]}]} 2023-07-14T18:23:35.270 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T18:23:35.271 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T18:23:35.271 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:23:35.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:35 smithi064 ceph-mon[96720]: pgmap v269: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2023-07-14T18:23:35.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:35 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/860615260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:23:35.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:35 smithi144 ceph-mon[98417]: pgmap v269: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2023-07-14T18:23:35.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:35 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/860615260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:23:35.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:35 smithi105 ceph-mon[98013]: pgmap v269: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 8.1 KiB/s wr, 20 op/s 2023-07-14T18:23:35.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:35 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/860615260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T18:23:35.836 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:37.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 -- 172.21.15.64:0/2111735537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 msgr2=0x7f32a010c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:37.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 --2- 172.21.15.64:0/2111735537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a010c6b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f329800b0a0 tx=0x7f329800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 -- 172.21.15.64:0/2111735537 shutdown_connections 2023-07-14T18:23:37.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 --2- 172.21.15.64:0/2111735537 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a010c6b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 --2- 172.21.15.64:0/2111735537 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0100f20 0x7f32a01012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 --2- 172.21.15.64:0/2111735537 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0101830 0x7f32a0109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 -- 172.21.15.64:0/2111735537 >> 172.21.15.64:0/2111735537 conn(0x7f32a0075ca0 msgr2=0x7f32a00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:37.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 -- 172.21.15.64:0/2111735537 shutdown_connections 2023-07-14T18:23:37.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.990+0000 7f32a6079700 1 -- 172.21.15.64:0/2111735537 wait complete. 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.991+0000 7f32a6079700 1 Processor -- start 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.991+0000 7f32a6079700 1 -- start start 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0100f20 0x7f32a0195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0101830 0x7f32a0196340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a019adc0 con 0x7f32a010a270 2023-07-14T18:23:37.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a019af30 con 0x7f32a0101830 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f32a6079700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32a019b0f0 con 0x7f32a0100f20 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47512/0 (socket says 172.21.15.64:47512) 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 -- 172.21.15.64:0/1199950616 learned_addr learned my addr 172.21.15.64:0/1199950616 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329effd700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0101830 0x7f32a0196340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329f7fe700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0100f20 0x7f32a0195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:37.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 -- 172.21.15.64:0/1199950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0100f20 msgr2=0x7f32a0195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0100f20 0x7f32a0195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 -- 172.21.15.64:0/1199950616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0101830 msgr2=0x7f32a0196340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0101830 0x7f32a0196340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 -- 172.21.15.64:0/1199950616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298009d00 con 0x7f32a010a270 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329ffff700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f329800b830 tx=0x7f32980094c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.992+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329800e040 con 0x7f32a010a270 2023-07-14T18:23:37.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.993+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3298003e90 con 0x7f32a010a270 2023-07-14T18:23:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.993+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a019b370 con 0x7f32a010a270 2023-07-14T18:23:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.993+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298004840 con 0x7f32a010a270 2023-07-14T18:23:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.993+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a01a2c30 con 0x7f32a010a270 2023-07-14T18:23:37.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.994+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32a0104b40 con 0x7f32a010a270 2023-07-14T18:23:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.995+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3298003a00 con 0x7f32a010a270 2023-07-14T18:23:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.995+0000 7f329cff9700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 0x7f328c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.995+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f32980ce340 con 0x7f32a010a270 2023-07-14T18:23:37.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.995+0000 7f329f7fe700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 0x7f328c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:36.996+0000 7f329f7fe700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 0x7f328c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3290005950 tx=0x7f329000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:37.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.000+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f32980977d0 con 0x7f32a010a270 2023-07-14T18:23:37.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.193+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32a0197260 con 0x7f32a010a270 2023-07-14T18:23:37.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.195+0000 7f329cff9700 1 -- 172.21.15.64:0/1199950616 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f329809adf0 con 0x7f32a010a270 2023-07-14T18:23:37.247 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:37.248 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:37.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 msgr2=0x7f328c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:37.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 0x7f328c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3290005950 tx=0x7f329000b410 comp rx=0 tx=0).stop 2023-07-14T18:23:37.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 msgr2=0x7f32a019a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f329800b830 tx=0x7f32980094c0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 shutdown_connections 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32a0100f20 0x7f32a0195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f328c0a11e0 0x7f328c0a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32a010a270 0x7f32a019a7b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 --2- 172.21.15.64:0/1199950616 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32a0101830 0x7f32a0196340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:37.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 >> 172.21.15.64:0/1199950616 conn(0x7f32a0075ca0 msgr2=0x7f32a0101c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:37.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 shutdown_connections 2023-07-14T18:23:37.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:37.197+0000 7f32a6079700 1 -- 172.21.15.64:0/1199950616 wait complete. 2023-07-14T18:23:37.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:37 smithi064 ceph-mon[96720]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 21 op/s 2023-07-14T18:23:37.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:37 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1199950616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:37.604 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T18:23:37.605 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T18:23:37.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:37 smithi144 ceph-mon[98417]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 21 op/s 2023-07-14T18:23:37.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:37 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1199950616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:37.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:37 smithi105 ceph-mon[98013]: pgmap v270: 129 pgs: 129 active+clean; 626 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 7.9 KiB/s wr, 21 op/s 2023-07-14T18:23:37.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:37 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1199950616' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:38.165 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:39.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 -- 172.21.15.64:0/3692181915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2410101830 msgr2=0x7f2410109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:39.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 --2- 172.21.15.64:0/3692181915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2410101830 0x7f2410109d30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f2400009990 tx=0x7f2400009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 -- 172.21.15.64:0/3692181915 shutdown_connections 2023-07-14T18:23:39.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 --2- 172.21.15.64:0/3692181915 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f241010a270 0x7f241010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 --2- 172.21.15.64:0/3692181915 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410100f20 0x7f24101012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 --2- 172.21.15.64:0/3692181915 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2410101830 0x7f2410109d30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 -- 172.21.15.64:0/3692181915 >> 172.21.15.64:0/3692181915 conn(0x7f2410075ca0 msgr2=0x7f24100760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:39.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 -- 172.21.15.64:0/3692181915 shutdown_connections 2023-07-14T18:23:39.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.291+0000 7f2414c5b700 1 -- 172.21.15.64:0/3692181915 wait complete. 2023-07-14T18:23:39.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 Processor -- start 2023-07-14T18:23:39.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 -- start start 2023-07-14T18:23:39.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2410100f20 0x7f2410197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:39.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410101830 0x7f24101984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:39.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:39.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241019cf70 con 0x7f241010a270 2023-07-14T18:23:39.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241019d0e0 con 0x7f2410101830 2023-07-14T18:23:39.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f2414c5b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241019d2a0 con 0x7f2410100f20 2023-07-14T18:23:39.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f240ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:39.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f240ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47524/0 (socket says 172.21.15.64:47524) 2023-07-14T18:23:39.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.292+0000 7f240ed9d700 1 -- 172.21.15.64:0/423081194 learned_addr learned my addr 172.21.15.64:0/423081194 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:39.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240e59c700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2410100f20 0x7f2410197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:39.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240dd9b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410101830 0x7f24101984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:39.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 -- 172.21.15.64:0/423081194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2410100f20 msgr2=0x7f2410197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:39.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2410100f20 0x7f2410197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 -- 172.21.15.64:0/423081194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410101830 msgr2=0x7f24101984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:39.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410101830 0x7f24101984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 -- 172.21.15.64:0/423081194 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000095f0 con 0x7f241010a270 2023-07-14T18:23:39.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f240ed9d700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2408007f10 tx=0x7f240800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:39.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408017070 con 0x7f241010a270 2023-07-14T18:23:39.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241019d580 con 0x7f241010a270 2023-07-14T18:23:39.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f240800f040 con 0x7f241010a270 2023-07-14T18:23:39.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408005710 con 0x7f241010a270 2023-07-14T18:23:39.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.293+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24101a4de0 con 0x7f241010a270 2023-07-14T18:23:39.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.294+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24101077f0 con 0x7f241010a270 2023-07-14T18:23:39.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.295+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2408005870 con 0x7f241010a270 2023-07-14T18:23:39.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.296+0000 7f23ff7fe700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 0x7f23f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:39.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.296+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f24080cc970 con 0x7f241010a270 2023-07-14T18:23:39.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.296+0000 7f240e59c700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 0x7f23f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:39.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.297+0000 7f240e59c700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 0x7f23f80a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2404005950 tx=0x7f240400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:39.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.300+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2408095d70 con 0x7f241010a270 2023-07-14T18:23:39.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.499+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f241004e340 con 0x7f241010a270 2023-07-14T18:23:39.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.500+0000 7f23ff7fe700 1 -- 172.21.15.64:0/423081194 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23947 (secure 0 0 0) 0x7f2408099390 con 0x7f241010a270 2023-07-14T18:23:39.553 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:23:39.553 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":85,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:23:09.852778+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:23:39.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 msgr2=0x7f23f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:39.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 0x7f23f80a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2404005950 tx=0x7f240400b410 comp rx=0 tx=0).stop 2023-07-14T18:23:39.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 msgr2=0x7f241019c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2408007f10 tx=0x7f240800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 shutdown_connections 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2410100f20 0x7f2410197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f23f80a11e0 0x7f23f80a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f241010a270 0x7f241019c960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 --2- 172.21.15.64:0/423081194 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2410101830 0x7f24101984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:39.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 >> 172.21.15.64:0/423081194 conn(0x7f2410075ca0 msgr2=0x7f24100fed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:39.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.502+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 shutdown_connections 2023-07-14T18:23:39.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:39.503+0000 7f2414c5b700 1 -- 172.21.15.64:0/423081194 wait complete. 2023-07-14T18:23:39.585 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:39 smithi064 ceph-mon[96720]: pgmap v271: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2023-07-14T18:23:39.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:39 smithi144 ceph-mon[98417]: pgmap v271: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2023-07-14T18:23:39.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:39 smithi105 ceph-mon[98013]: pgmap v271: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2023-07-14T18:23:39.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T18:23:39.932 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T18:23:39.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T18:23:39.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T18:23:39.934 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T18:23:39.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T18:23:39.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T18:23:39.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T18:23:39.936 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T18:23:39.937 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T18:23:39.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T18:23:39.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T18:23:40.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:40 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/423081194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:40.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:40 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/423081194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:40.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:40 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/423081194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:23:40.937 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:41.074 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:41.397 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:41 smithi064 ceph-mon[96720]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 22 op/s 2023-07-14T18:23:41.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:41 smithi144 ceph-mon[98417]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 22 op/s 2023-07-14T18:23:41.761 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:41.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:41 smithi105 ceph-mon[98013]: pgmap v272: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 22 op/s 2023-07-14T18:23:41.904 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:41.920 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:42.197 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:42.371 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:42.785 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:42.787 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:43.166 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:43.167 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:43.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:43 smithi144 ceph-mon[98417]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2023-07-14T18:23:43.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:43 smithi064 ceph-mon[96720]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2023-07-14T18:23:43.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:43 smithi105 ceph-mon[98013]: pgmap v273: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.2 KiB/s wr, 15 op/s 2023-07-14T18:23:44.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 -- 172.21.15.64:0/1132715999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7140ff7e0 msgr2=0x7fa7141037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 --2- 172.21.15.64:0/1132715999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7140ff7e0 0x7fa7141037b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa7080099c0 tx=0x7fa708009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 -- 172.21.15.64:0/1132715999 shutdown_connections 2023-07-14T18:23:44.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 --2- 172.21.15.64:0/1132715999 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa7140fee40 0x7fa7140ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 --2- 172.21.15.64:0/1132715999 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa714103ee0 0x7fa71410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 --2- 172.21.15.64:0/1132715999 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7140ff7e0 0x7fa7141037b0 secure :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa7080099c0 tx=0x7fa708009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 -- 172.21.15.64:0/1132715999 >> 172.21.15.64:0/1132715999 conn(0x7fa714075ca0 msgr2=0x7fa7140760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 -- 172.21.15.64:0/1132715999 shutdown_connections 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.164+0000 7fa719296700 1 -- 172.21.15.64:0/1132715999 wait complete. 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 Processor -- start 2023-07-14T18:23:44.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 -- start start 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7140fee40 0x7fa714196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa714103ee0 0x7fa714197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 0x7fa7141a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141a2640 con 0x7fa7141979b0 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141a27b0 con 0x7fa7140fee40 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa719296700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7141a2970 con 0x7fa714103ee0 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa7137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 0x7fa7141a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:44.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa7137fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 0x7fa7141a1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47536/0 (socket says 172.21.15.64:47536) 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa7137fe700 1 -- 172.21.15.64:0/98320624 learned_addr learned my addr 172.21.15.64:0/98320624 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa712ffd700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7140fee40 0x7fa714196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.165+0000 7fa7127fc700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa714103ee0 0x7fa714197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 -- 172.21.15.64:0/98320624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa714103ee0 msgr2=0x7fa714197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa714103ee0 0x7fa714197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 -- 172.21.15.64:0/98320624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7140fee40 msgr2=0x7fa714196dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa7140fee40 0x7fa714196dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7080095f0 con 0x7fa7141979b0 2023-07-14T18:23:44.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7127fc700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa714103ee0 0x7fa714197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:44.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa7137fe700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 0x7fa7141a1e70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa70c00ab50 tx=0x7fa70c00ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:44.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c00f070 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa70c007a80 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7141a2c50 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70c005280 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.166+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7141a3100 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.167+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa71410ec10 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.168+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa70c004ad0 con 0x7fa7141979b0 2023-07-14T18:23:44.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa6fbfff700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa6fc0a11e0 0x7fa6fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa712ffd700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa6fc0a11e0 0x7fa6fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa70c0ccbc0 con 0x7fa7141979b0 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa6fbfff700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] conn(0x7fa6fc0abd50 0x7fa6fc0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6fc0ae850 con 0x7fa6fc0abd50 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa70c0ccf30 con 0x7fa7141979b0 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa7127fc700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] conn(0x7fa6fc0abd50 0x7fa6fc0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:44.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa712ffd700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa6fc0a11e0 0x7fa6fc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa704009a10 tx=0x7fa704009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.169+0000 7fa7127fc700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] conn(0x7fa6fc0abd50 0x7fa6fc0ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.170+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== osd.6 v2:172.21.15.64:6818/394567036 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa6fc0ae850 con 0x7fa6fc0abd50 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.186+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 --> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa71404e340 con 0x7fa6fc0abd50 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.186+0000 7fa6fbfff700 1 -- 172.21.15.64:0/98320624 <== osd.6 v2:172.21.15.64:6818/394567036 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa71404e340 con 0x7fa6fc0abd50 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] conn(0x7fa6fc0abd50 msgr2=0x7fa6fc0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] conn(0x7fa6fc0abd50 0x7fa6fc0ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa6fc0a11e0 msgr2=0x7fa6fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa6fc0a11e0 0x7fa6fc0a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa704009a10 tx=0x7fa704009190 comp rx=0 tx=0).stop 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 msgr2=0x7fa7141a1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 --2- 172.21.15.64:0/98320624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa7141979b0 0x7fa7141a1e70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa70c00ab50 tx=0x7fa70c00ae60 comp rx=0 tx=0).stop 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa7137fe700 1 -- 172.21.15.64:0/98320624 reap_dead start 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 shutdown_connections 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 >> 172.21.15.64:0/98320624 conn(0x7fa714075ca0 msgr2=0x7fa7140fdc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 shutdown_connections 2023-07-14T18:23:44.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:44.187+0000 7fa719296700 1 -- 172.21.15.64:0/98320624 wait complete. 2023-07-14T18:23:45.591 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:45 smithi064 ceph-mon[96720]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2023-07-14T18:23:45.631 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:45.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:45 smithi144 ceph-mon[98417]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2023-07-14T18:23:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- 172.21.15.64:0/2122692869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf4103ee0 msgr2=0x7fddf410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 --2- 172.21.15.64:0/2122692869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf4103ee0 0x7fddf410ba60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fddec00b0d0 tx=0x7fddec00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- 172.21.15.64:0/2122692869 shutdown_connections 2023-07-14T18:23:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 --2- 172.21.15.64:0/2122692869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf4103ee0 0x7fddf410ba60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.725 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 --2- 172.21.15.64:0/2122692869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddf40fee40 0x7fddf40ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 --2- 172.21.15.64:0/2122692869 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf40ff7e0 0x7fddf41037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- 172.21.15.64:0/2122692869 >> 172.21.15.64:0/2122692869 conn(0x7fddf4075ca0 msgr2=0x7fddf40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- 172.21.15.64:0/2122692869 shutdown_connections 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- 172.21.15.64:0/2122692869 wait complete. 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 Processor -- start 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.557+0000 7fddfb900700 1 -- start start 2023-07-14T18:23:45.726 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 0x7fddf4194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddf40ff7e0 0x7fddf41952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf4103ee0 0x7fddf4199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4199d30 con 0x7fddf40fee40 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf4199ea0 con 0x7fddf40ff7e0 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf419a060 con 0x7fddf4103ee0 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 0x7fddf4194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:45.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 0x7fddf4194d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47548/0 (socket says 172.21.15.64:47548) 2023-07-14T18:23:45.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 -- 172.21.15.64:0/1386826135 learned_addr learned my addr 172.21.15.64:0/1386826135 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:45.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf8e9b700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddf40ff7e0 0x7fddf41952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:45.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf9e9d700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf4103ee0 0x7fddf4199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:45.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 -- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf4103ee0 msgr2=0x7fddf4199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf4103ee0 0x7fddf4199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 -- 172.21.15.64:0/1386826135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddf40ff7e0 msgr2=0x7fddf41952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fddf40ff7e0 0x7fddf41952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddec009d00 con 0x7fddf40fee40 2023-07-14T18:23:45.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf9e9d700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fddf4103ee0 0x7fddf4199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:45.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddf969c700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 0x7fddf4194d70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fddf000d980 tx=0x7fddf000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:45.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0004220 con 0x7fddf40fee40 2023-07-14T18:23:45.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fddf0007640 con 0x7fddf40fee40 2023-07-14T18:23:45.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf419a340 con 0x7fddf40fee40 2023-07-14T18:23:45.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf00055e0 con 0x7fddf40fee40 2023-07-14T18:23:45.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.558+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf41a1ba0 con 0x7fddf40fee40 2023-07-14T18:23:45.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.559+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fddf410ed20 con 0x7fddf40fee40 2023-07-14T18:23:45.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.561+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fddf0004750 con 0x7fddf40fee40 2023-07-14T18:23:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.561+0000 7fddea7fc700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdde00a1260 0x7fdde00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.561+0000 7fddf8e9b700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdde00a1260 0x7fdde00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.561+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fddf00cd0a0 con 0x7fddf40fee40 2023-07-14T18:23:45.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddea7fc700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] conn(0x7fdde00abdd0 0x7fdde00ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:45.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdde00ae8d0 con 0x7fdde00abdd0 2023-07-14T18:23:45.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddf00cd4b0 con 0x7fddf40fee40 2023-07-14T18:23:45.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddf8e9b700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdde00a1260 0x7fdde00a3710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fddf4196390 tx=0x7fdde4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:45.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddf9e9d700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] conn(0x7fdde00abdd0 0x7fdde00ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:45.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.562+0000 7fddf9e9d700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] conn(0x7fdde00abdd0 0x7fdde00ae200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:45.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.563+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== osd.5 v2:172.21.15.144:6808/681164415 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fdde00ae8d0 con 0x7fdde00abdd0 2023-07-14T18:23:45.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.581+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 --> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fddf404e340 con 0x7fdde00abdd0 2023-07-14T18:23:45.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddea7fc700 1 -- 172.21.15.64:0/1386826135 <== osd.5 v2:172.21.15.144:6808/681164415 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fddf404e340 con 0x7fdde00abdd0 2023-07-14T18:23:45.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] conn(0x7fdde00abdd0 msgr2=0x7fdde00ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] conn(0x7fdde00abdd0 0x7fdde00ae200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:45.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdde00a1260 msgr2=0x7fdde00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdde00a1260 0x7fdde00a3710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fddf4196390 tx=0x7fdde4009190 comp rx=0 tx=0).stop 2023-07-14T18:23:45.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 msgr2=0x7fddf4194d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:45.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.582+0000 7fddfb900700 1 --2- 172.21.15.64:0/1386826135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddf40fee40 0x7fddf4194d70 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fddf000d980 tx=0x7fddf000dc90 comp rx=0 tx=0).stop 2023-07-14T18:23:45.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.583+0000 7fddf9e9d700 1 -- 172.21.15.64:0/1386826135 reap_dead start 2023-07-14T18:23:45.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.584+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 shutdown_connections 2023-07-14T18:23:45.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.585+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 >> 172.21.15.64:0/1386826135 conn(0x7fddf4075ca0 msgr2=0x7fddf40fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:45.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.585+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 shutdown_connections 2023-07-14T18:23:45.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:45.585+0000 7fddfb900700 1 -- 172.21.15.64:0/1386826135 wait complete. 2023-07-14T18:23:45.739 INFO:teuthology.orchestra.run.smithi064.stdout:193273528360 2023-07-14T18:23:45.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T18:23:45.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:45 smithi105 ceph-mon[98013]: pgmap v274: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.7 KiB/s wr, 14 op/s 2023-07-14T18:23:46.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 -- 172.21.15.64:0/1538883465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f80fffb0 msgr2=0x7f22f8100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 --2- 172.21.15.64:0/1538883465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f80fffb0 0x7f22f8100380 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f22e80099c0 tx=0x7f22e8009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 -- 172.21.15.64:0/1538883465 shutdown_connections 2023-07-14T18:23:46.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 --2- 172.21.15.64:0/1538883465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 0x7f22f810cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 --2- 172.21.15.64:0/1538883465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f80fffb0 0x7f22f8100380 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 --2- 172.21.15.64:0/1538883465 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22f8100950 0x7f22f81048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.048+0000 7f22fd850700 1 -- 172.21.15.64:0/1538883465 >> 172.21.15.64:0/1538883465 conn(0x7f22f8077cd0 msgr2=0x7f22f80780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:46.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.049+0000 7f22fd850700 1 -- 172.21.15.64:0/1538883465 shutdown_connections 2023-07-14T18:23:46.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.049+0000 7f22fd850700 1 -- 172.21.15.64:0/1538883465 wait complete. 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.049+0000 7f22fd850700 1 Processor -- start 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.049+0000 7f22fd850700 1 -- start start 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22f80fffb0 0x7f22f819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 0x7f22f819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 0x7f22f8195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81961c0 con 0x7f22f8100950 2023-07-14T18:23:46.105 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f8196330 con 0x7f22f8105000 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f81964f0 con 0x7f22f80fffb0 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 0x7f22f819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 0x7f22f819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47566/0 (socket says 172.21.15.64:47566) 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 -- 172.21.15.64:0/4055684544 learned_addr learned my addr 172.21.15.64:0/4055684544 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f6ffd700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22f80fffb0 0x7f22f819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f77fe700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 0x7f22f8195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:46.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 -- 172.21.15.64:0/4055684544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22f80fffb0 msgr2=0x7f22f819bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f22f80fffb0 0x7f22f819bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 -- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 msgr2=0x7f22f8195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 0x7f22f8195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e80095f0 con 0x7f22f8100950 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f77fe700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f22f8105000 0x7f22f8195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22f67fc700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 0x7f22f819c010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00e970 tx=0x7f22ec00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:46.107 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec009790 con 0x7f22f8100950 2023-07-14T18:23:46.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f81967d0 con 0x7f22f8100950 2023-07-14T18:23:46.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f22ec00eec0 con 0x7f22f8100950 2023-07-14T18:23:46.108 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.050+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0054b0 con 0x7f22f8100950 2023-07-14T18:23:46.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.051+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f81a2ac0 con 0x7f22f8100950 2023-07-14T18:23:46.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.052+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22f804e340 con 0x7f22f8100950 2023-07-14T18:23:46.109 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.054+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f22ec005610 con 0x7f22f8100950 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22fc84e700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f22e00a11e0 0x7f22e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22f6ffd700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f22e00a11e0 0x7f22e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f22ec014070 con 0x7f22f8100950 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22fc84e700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] conn(0x7f22e00abd50 0x7f22e00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22e00ae850 con 0x7f22e00abd50 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.055+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22ec0cd540 con 0x7f22f8100950 2023-07-14T18:23:46.110 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.056+0000 7f22f6ffd700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f22e00a11e0 0x7f22e00a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009990 tx=0x7f22e8005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:46.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.056+0000 7f22f77fe700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] conn(0x7f22e00abd50 0x7f22e00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:46.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.056+0000 7f22f77fe700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] conn(0x7f22e00abd50 0x7f22e00ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:46.111 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.057+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== osd.1 v2:172.21.15.105:6800/1803665437 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f22e00ae850 con 0x7f22e00abd50 2023-07-14T18:23:46.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.073+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 --> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f22f810fdc0 con 0x7f22e00abd50 2023-07-14T18:23:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.073+0000 7f22fc84e700 1 -- 172.21.15.64:0/4055684544 <== osd.1 v2:172.21.15.105:6800/1803665437 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f22f0009940 con 0x7f22e00abd50 2023-07-14T18:23:46.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] conn(0x7f22e00abd50 msgr2=0x7f22e00ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] conn(0x7f22e00abd50 0x7f22e00ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f22e00a11e0 msgr2=0x7f22e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f22e00a11e0 0x7f22e00a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f22e8009990 tx=0x7f22e8005c00 comp rx=0 tx=0).stop 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 msgr2=0x7f22f819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 --2- 172.21.15.64:0/4055684544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22f8100950 0x7f22f819c010 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00e970 tx=0x7f22ec00ec80 comp rx=0 tx=0).stop 2023-07-14T18:23:46.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22f77fe700 1 -- 172.21.15.64:0/4055684544 reap_dead start 2023-07-14T18:23:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 shutdown_connections 2023-07-14T18:23:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 >> 172.21.15.64:0/4055684544 conn(0x7f22f8077cd0 msgr2=0x7f22f80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 shutdown_connections 2023-07-14T18:23:46.127 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:46.074+0000 7f22fd850700 1 -- 172.21.15.64:0/4055684544 wait complete. 2023-07-14T18:23:47.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 -- 172.21.15.64:0/3814875425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 msgr2=0x7fe6fc10cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 --2- 172.21.15.64:0/3814875425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc10cb30 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400b0a0 tx=0x7fe6f400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 -- 172.21.15.64:0/3814875425 shutdown_connections 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 --2- 172.21.15.64:0/3814875425 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6fc104c70 0x7fe6fc105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 --2- 172.21.15.64:0/3814875425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc10cb30 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 --2- 172.21.15.64:0/3814875425 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc0fe210 0x7fe6fc0fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 -- 172.21.15.64:0/3814875425 >> 172.21.15.64:0/3814875425 conn(0x7fe6fc075280 msgr2=0x7fe6fc075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 -- 172.21.15.64:0/3814875425 shutdown_connections 2023-07-14T18:23:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 -- 172.21.15.64:0/3814875425 wait complete. 2023-07-14T18:23:47.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.113+0000 7fe701cec700 1 Processor -- start 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 -- start start 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6fc0fe210 0x7fe6fc0ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc0ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104c70 0x7fe6fc1a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc100220 con 0x7fe6fc0febc0 2023-07-14T18:23:47.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc1a44c0 con 0x7fe6fc104c70 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe701cec700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc1a4630 con 0x7fe6fc0fe210 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc0ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc0ff930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47580/0 (socket says 172.21.15.64:47580) 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 -- 172.21.15.64:0/3263003250 learned_addr learned my addr 172.21.15.64:0/3263003250 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6fb7fe700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6fc0fe210 0x7fe6fc0ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6fbfff700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104c70 0x7fe6fc1a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 -- 172.21.15.64:0/3263003250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6fc0fe210 msgr2=0x7fe6fc0ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe6fc0fe210 0x7fe6fc0ff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 -- 172.21.15.64:0/3263003250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104c70 msgr2=0x7fe6fc1a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe6fc104c70 0x7fe6fc1a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.114+0000 7fe6faffd700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f4009d00 con 0x7fe6fc0febc0 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe6faffd700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc0ff930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d950 tx=0x7fe6ec00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec00f890 con 0x7fe6fc0febc0 2023-07-14T18:23:47.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe6ec00fe80 con 0x7fe6fc0febc0 2023-07-14T18:23:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc1a48b0 con 0x7fe6fc0febc0 2023-07-14T18:23:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec00e3b0 con 0x7fe6fc0febc0 2023-07-14T18:23:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.115+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc1a4e00 con 0x7fe6fc0febc0 2023-07-14T18:23:47.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.116+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6fc10fda0 con 0x7fe6fc0febc0 2023-07-14T18:23:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.118+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe6ec010460 con 0x7fe6fc0febc0 2023-07-14T18:23:47.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6f8ff9700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6e80a1260 0x7fe6e80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6fb7fe700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6e80a1260 0x7fe6e80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe6ec0ccef0 con 0x7fe6fc0febc0 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6f8ff9700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] conn(0x7fe6e80abdd0 0x7fe6e80ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6e80ae8d0 con 0x7fe6e80abdd0 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6ec0cd300 con 0x7fe6fc0febc0 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6fbfff700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] conn(0x7fe6e80abdd0 0x7fe6e80ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:47.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6fb7fe700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6e80a1260 0x7fe6e80a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009be0 tx=0x7fe6e4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.119+0000 7fe6fbfff700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] conn(0x7fe6e80abdd0 0x7fe6e80ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:47.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.120+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== osd.9 v2:172.21.15.64:6826/2478630818 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fe6e80ae8d0 con 0x7fe6e80abdd0 2023-07-14T18:23:47.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 --> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6fc04e340 con 0x7fe6e80abdd0 2023-07-14T18:23:47.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe6f8ff9700 1 -- 172.21.15.64:0/3263003250 <== osd.9 v2:172.21.15.64:6826/2478630818 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe6fc04e340 con 0x7fe6e80abdd0 2023-07-14T18:23:47.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] conn(0x7fe6e80abdd0 msgr2=0x7fe6e80ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] conn(0x7fe6e80abdd0 0x7fe6e80ae200 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6e80a1260 msgr2=0x7fe6e80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe6e80a1260 0x7fe6e80a3710 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009be0 tx=0x7fe6e4009260 comp rx=0 tx=0).stop 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 msgr2=0x7fe6fc0ff930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe701cec700 1 --2- 172.21.15.64:0/3263003250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6fc0febc0 0x7fe6fc0ff930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d950 tx=0x7fe6ec00dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.136+0000 7fe6fbfff700 1 -- 172.21.15.64:0/3263003250 reap_dead start 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.137+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 shutdown_connections 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.137+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 >> 172.21.15.64:0/3263003250 conn(0x7fe6fc075280 msgr2=0x7fe6fc0fcaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:47.189 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.137+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 shutdown_connections 2023-07-14T18:23:47.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.137+0000 7fe701cec700 1 -- 172.21.15.64:0/3263003250 wait complete. 2023-07-14T18:23:47.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:47 smithi144 ceph-mon[98417]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T18:23:47.684 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:47 smithi064 ceph-mon[96720]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T18:23:47.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:47 smithi105 ceph-mon[98013]: pgmap v275: 129 pgs: 129 active+clean; 636 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2023-07-14T18:23:48.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 -- 172.21.15.64:0/3125074809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 msgr2=0x7f443c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 --2- 172.21.15.64:0/3125074809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c1037b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4430009920 tx=0x7f4430009c30 comp rx=0 tx=0).stop 2023-07-14T18:23:48.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 -- 172.21.15.64:0/3125074809 shutdown_connections 2023-07-14T18:23:48.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 --2- 172.21.15.64:0/3125074809 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 0x7f443c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 --2- 172.21.15.64:0/3125074809 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 0x7f443c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 --2- 172.21.15.64:0/3125074809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c1037b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 -- 172.21.15.64:0/3125074809 >> 172.21.15.64:0/3125074809 conn(0x7f443c075ca0 msgr2=0x7f443c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 -- 172.21.15.64:0/3125074809 shutdown_connections 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.855+0000 7f44419df700 1 -- 172.21.15.64:0/3125074809 wait complete. 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 Processor -- start 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 -- start start 2023-07-14T18:23:48.407 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 0x7f443c196c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 0x7f443c1a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c197a70 con 0x7f443c0ff7e0 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1a21e0 con 0x7f443c103ee0 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f44419df700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1a2350 con 0x7f443c0fee40 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.856+0000 7f443a7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:48.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c197160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47596/0 (socket says 172.21.15.64:47596) 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 -- 172.21.15.64:0/4228744137 learned_addr learned my addr 172.21.15.64:0/4228744137 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443affd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 0x7f443c196c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443b7fe700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 0x7f443c1a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 -- 172.21.15.64:0/4228744137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 msgr2=0x7f443c196c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 0x7f443c196c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 -- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 msgr2=0x7f443c1a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 0x7f443c1a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434009d00 con 0x7f443c0ff7e0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443affd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f443c0fee40 0x7f443c196c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443b7fe700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f443c103ee0 0x7f443c1a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f443a7fc700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c197160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4430009230 tx=0x7f443000fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443001c070 con 0x7f443c0ff7e0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4430005040 con 0x7f443c0ff7e0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4430009520 con 0x7f443c0ff7e0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44300039e0 con 0x7f443c0ff7e0 2023-07-14T18:23:48.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.857+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c1a2850 con 0x7f443c0ff7e0 2023-07-14T18:23:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.858+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f443c04e340 con 0x7f443c0ff7e0 2023-07-14T18:23:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.859+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4430020030 con 0x7f443c0ff7e0 2023-07-14T18:23:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f44409dd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f44240a11e0 0x7f44240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f443affd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f44240a11e0 0x7f44240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:48.411 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f44300ce060 con 0x7f443c0ff7e0 2023-07-14T18:23:48.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f44409dd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] conn(0x7f44240abd50 0x7f44240ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:48.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44240ae850 con 0x7f44240abd50 2023-07-14T18:23:48.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44300ce3d0 con 0x7f443c0ff7e0 2023-07-14T18:23:48.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f443affd700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f44240a11e0 0x7f44240a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f442c009cb0 tx=0x7f442c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:48.412 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.860+0000 7f443b7fe700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] conn(0x7f44240abd50 0x7f44240ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:48.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.861+0000 7f443b7fe700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] conn(0x7f44240abd50 0x7f44240ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:48.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.862+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== osd.7 v2:172.21.15.105:6816/554937712 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f44240ae850 con 0x7f44240abd50 2023-07-14T18:23:48.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.877+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 --> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f443c10ec30 con 0x7f44240abd50 2023-07-14T18:23:48.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.878+0000 7f44409dd700 1 -- 172.21.15.64:0/4228744137 <== osd.7 v2:172.21.15.105:6816/554937712 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f443400f040 con 0x7f44240abd50 2023-07-14T18:23:48.413 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.878+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] conn(0x7f44240abd50 msgr2=0x7f44240ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.878+0000 7f44419df700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] conn(0x7f44240abd50 0x7f44240ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f44240a11e0 msgr2=0x7f44240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f44240a11e0 0x7f44240a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f442c009cb0 tx=0x7f442c009260 comp rx=0 tx=0).stop 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 msgr2=0x7f443c197160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 --2- 172.21.15.64:0/4228744137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f443c0ff7e0 0x7f443c197160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4430009230 tx=0x7f443000fa00 comp rx=0 tx=0).stop 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f443b7fe700 1 -- 172.21.15.64:0/4228744137 reap_dead start 2023-07-14T18:23:48.414 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 shutdown_connections 2023-07-14T18:23:48.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 >> 172.21.15.64:0/4228744137 conn(0x7f443c075ca0 msgr2=0x7f443c0fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:48.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 shutdown_connections 2023-07-14T18:23:48.415 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:47.879+0000 7f44419df700 1 -- 172.21.15.64:0/4228744137 wait complete. 2023-07-14T18:23:48.721 INFO:teuthology.orchestra.run.smithi064.stdout:163208757293 2023-07-14T18:23:48.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T18:23:48.768 INFO:teuthology.orchestra.run.smithi064.stdout:274877906972 2023-07-14T18:23:48.769 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T18:23:48.929 INFO:teuthology.orchestra.run.smithi064.stdout:55834574910 2023-07-14T18:23:48.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T18:23:49.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2570940718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac0ff7d0 msgr2=0x7fc6ac1037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2570940718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac1037a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc6940099c0 tx=0x7fc694009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2570940718 shutdown_connections 2023-07-14T18:23:49.274 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2570940718 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac10ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2570940718 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 0x7fc6ac0ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2570940718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac1037a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2570940718 >> 172.21.15.64:0/2570940718 conn(0x7fc6ac0fa6d0 msgr2=0x7fc6ac0fcae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2570940718 shutdown_connections 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2570940718 wait complete. 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.121+0000 7fc6b1a0f700 1 Processor -- start 2023-07-14T18:23:49.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 -- start start 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 0x7fc6ac100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1a41e0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1a4350 con 0x7fc6ac0fee30 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6b1a0f700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1a4510 con 0x7fc6ac0ff7d0 2023-07-14T18:23:49.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6ab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6ab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac101300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47610/0 (socket says 172.21.15.64:47610) 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6ab7fe700 1 -- 172.21.15.64:0/2699519058 learned_addr learned my addr 172.21.15.64:0/2699519058 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6aaffd700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 0x7fc6ac100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.122+0000 7fc6ab7fe700 1 -- 172.21.15.64:0/2699519058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0ff7d0 msgr2=0x7fc6ac100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6aa7fc700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6ab7fe700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac100d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.277 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6ab7fe700 1 -- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 msgr2=0x7fc6ac100840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6ab7fe700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 0x7fc6ac100840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6ab7fe700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940095f0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6aa7fc700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6ac0ff7d0 0x7fc6ac100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6aaffd700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6ac0fee30 0x7fc6ac100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6ab7fe700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac101300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400e940 tx=0x7fc6a400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40097b0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac1a47f0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6a400ee90 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4005450 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.123+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac1a4cf0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.124+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6ac10ec70 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.126+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc6a40055b0 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6b0a0d700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6980a11e0 0x7fc6980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc6a4014070 con 0x7fc6ac103ed0 2023-07-14T18:23:49.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6aaffd700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6980a11e0 0x7fc6980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6b0a0d700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] conn(0x7fc6980abd50 0x7fc6980ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6980ae850 con 0x7fc6980abd50 2023-07-14T18:23:49.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.127+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc6a40cd460 con 0x7fc6ac103ed0 2023-07-14T18:23:49.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.128+0000 7fc6aa7fc700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] conn(0x7fc6980abd50 0x7fc6980ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.128+0000 7fc6aaffd700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6980a11e0 0x7fc6980a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c0091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.128+0000 7fc6aa7fc700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] conn(0x7fc6980abd50 0x7fc6980ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.129+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== osd.4 v2:172.21.15.105:6808/1784745142 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc6980ae850 con 0x7fc6980abd50 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.148+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 --> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6ac04e340 con 0x7fc6980abd50 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.149+0000 7fc6b0a0d700 1 -- 172.21.15.64:0/2699519058 <== osd.4 v2:172.21.15.105:6808/1784745142 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc6ac04e340 con 0x7fc6980abd50 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.149+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] conn(0x7fc6980abd50 msgr2=0x7fc6980ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.281 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.149+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] conn(0x7fc6980abd50 0x7fc6980ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6980a11e0 msgr2=0x7fc6980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6980a11e0 0x7fc6980a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc69c009fd0 tx=0x7fc69c0091b0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 msgr2=0x7fc6ac101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 --2- 172.21.15.64:0/2699519058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6ac103ed0 0x7fc6ac101300 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc6a400e940 tx=0x7fc6a400ec50 comp rx=0 tx=0).stop 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6ab7fe700 1 -- 172.21.15.64:0/2699519058 reap_dead start 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 shutdown_connections 2023-07-14T18:23:49.282 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 >> 172.21.15.64:0/2699519058 conn(0x7fc6ac0fa6d0 msgr2=0x7fc6ac068ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 shutdown_connections 2023-07-14T18:23:49.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.150+0000 7fc6b1a0f700 1 -- 172.21.15.64:0/2699519058 wait complete. 2023-07-14T18:23:49.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2616401443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e64104fd0 msgr2=0x7f5e6410cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2616401443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e64104fd0 0x7f5e6410cb50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f5e5c00b0d0 tx=0x7f5e5c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2616401443 shutdown_connections 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2616401443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e640fff30 0x7f5e64100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2616401443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e64104fd0 0x7f5e6410cb50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2616401443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e641008d0 0x7f5e641048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2616401443 >> 172.21.15.64:0/2616401443 conn(0x7f5e640fb790 msgr2=0x7f5e640fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2616401443 shutdown_connections 2023-07-14T18:23:49.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.389+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2616401443 wait complete. 2023-07-14T18:23:49.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 Processor -- start 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 -- start start 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e640fff30 0x7f5e64072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 0x7f5e64072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e64104fd0 0x7f5e64073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e641a42a0 con 0x7f5e641008d0 2023-07-14T18:23:49.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e641a4410 con 0x7f5e640fff30 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e6b64e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e641a45d0 con 0x7f5e64104fd0 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e68be9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 0x7f5e64072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e68be9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 0x7f5e64072e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47624/0 (socket says 172.21.15.64:47624) 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e68be9700 1 -- 172.21.15.64:0/2137787834 learned_addr learned my addr 172.21.15.64:0/2137787834 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e69beb700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e64104fd0 0x7f5e64073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.390+0000 7f5e693ea700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e640fff30 0x7f5e64072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 -- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e64104fd0 msgr2=0x7f5e64073400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5e64104fd0 0x7f5e64073400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 -- 172.21.15.64:0/2137787834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e640fff30 msgr2=0x7f5e64072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e640fff30 0x7f5e64072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c009d00 con 0x7f5e641008d0 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e68be9700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 0x7f5e64072e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000d980 tx=0x7f5e6000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60009930 con 0x7f5e641008d0 2023-07-14T18:23:49.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5e60010460 con 0x7f5e641008d0 2023-07-14T18:23:49.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e641a48b0 con 0x7f5e641008d0 2023-07-14T18:23:49.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6000f620 con 0x7f5e641008d0 2023-07-14T18:23:49.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.391+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e641a4db0 con 0x7f5e641008d0 2023-07-14T18:23:49.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.392+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e6410fce0 con 0x7f5e641008d0 2023-07-14T18:23:49.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.393+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5e600105d0 con 0x7f5e641008d0 2023-07-14T18:23:49.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e5a7fc700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5e500a11e0 0x7f5e500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e693ea700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5e500a11e0 0x7f5e500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5e600ccaa0 con 0x7f5e641008d0 2023-07-14T18:23:49.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e5a7fc700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] conn(0x7f5e500abd50 0x7f5e500ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e500ae850 con 0x7f5e500abd50 2023-07-14T18:23:49.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e600cceb0 con 0x7f5e641008d0 2023-07-14T18:23:49.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e693ea700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5e500a11e0 0x7f5e500a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e54005fd0 tx=0x7f5e5400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.394+0000 7f5e69beb700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] conn(0x7f5e500abd50 0x7f5e500ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.395+0000 7f5e69beb700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] conn(0x7f5e500abd50 0x7f5e500ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.396+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== osd.8 v2:172.21.15.144:6816/1466260648 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f5e500ae850 con 0x7f5e500abd50 2023-07-14T18:23:49.462 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.411+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 --> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e6404e340 con 0x7f5e500abd50 2023-07-14T18:23:49.463 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e5a7fc700 1 -- 172.21.15.64:0/2137787834 <== osd.8 v2:172.21.15.144:6816/1466260648 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e6404e340 con 0x7f5e500abd50 2023-07-14T18:23:49.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] conn(0x7f5e500abd50 msgr2=0x7f5e500ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] conn(0x7f5e500abd50 0x7f5e500ae180 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.464 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5e500a11e0 msgr2=0x7f5e500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5e500a11e0 0x7f5e500a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5e54005fd0 tx=0x7f5e5400a560 comp rx=0 tx=0).stop 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 msgr2=0x7f5e64072e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e6b64e700 1 --2- 172.21.15.64:0/2137787834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5e641008d0 0x7f5e64072e80 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000d980 tx=0x7f5e6000dc90 comp rx=0 tx=0).stop 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.412+0000 7f5e69beb700 1 -- 172.21.15.64:0/2137787834 reap_dead start 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.413+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 shutdown_connections 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.413+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 >> 172.21.15.64:0/2137787834 conn(0x7f5e640fb790 msgr2=0x7f5e640fc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.465 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.413+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 shutdown_connections 2023-07-14T18:23:49.466 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.413+0000 7f5e6b64e700 1 -- 172.21.15.64:0/2137787834 wait complete. 2023-07-14T18:23:49.647 INFO:teuthology.orchestra.run.smithi064.stdout:214748364836 2023-07-14T18:23:49.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T18:23:49.676 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:49 smithi064 ceph-mon[96720]: pgmap v276: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2023-07-14T18:23:49.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:49 smithi144 ceph-mon[98417]: pgmap v276: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2023-07-14T18:23:49.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:49 smithi105 ceph-mon[98013]: pgmap v276: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2023-07-14T18:23:49.829 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.718+0000 7ff986d40700 1 -- 172.21.15.64:0/2700283292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 msgr2=0x7ff980105040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 --2- 172.21.15.64:0/2700283292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff980105040 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff974009990 tx=0x7ff974009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 -- 172.21.15.64:0/2700283292 shutdown_connections 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 --2- 172.21.15.64:0/2700283292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9800febc0 0x7ff98010cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 --2- 172.21.15.64:0/2700283292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff980105040 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 --2- 172.21.15.64:0/2700283292 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9800fe210 0x7ff9800fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 -- 172.21.15.64:0/2700283292 >> 172.21.15.64:0/2700283292 conn(0x7ff980075280 msgr2=0x7ff980075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 -- 172.21.15.64:0/2700283292 shutdown_connections 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 -- 172.21.15.64:0/2700283292 wait complete. 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 Processor -- start 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.719+0000 7ff986d40700 1 -- start start 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9800fe210 0x7ff9800ff3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9800febc0 0x7ff9800ff930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff9801a3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff980100220 con 0x7ff980104c70 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9801a43a0 con 0x7ff9800fe210 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff986d40700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9801a4510 con 0x7ff9800febc0 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff9801a3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff97ffff700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9800febc0 0x7ff9800ff930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff9801a3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47642/0 (socket says 172.21.15.64:47642) 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 -- 172.21.15.64:0/2916957845 learned_addr learned my addr 172.21.15.64:0/2916957845 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff984adc700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9800fe210 0x7ff9800ff3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 -- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9800febc0 msgr2=0x7ff9800ff930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff9800febc0 0x7ff9800ff930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 -- 172.21.15.64:0/2916957845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9800fe210 msgr2=0x7ff9800ff3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff9800fe210 0x7ff9800ff3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9740095f0 con 0x7ff980104c70 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff9852dd700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff9801a3e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff978007d70 tx=0x7ff97800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.720+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97800ad40 con 0x7ff980104c70 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.721+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff9780128b0 con 0x7ff980104c70 2023-07-14T18:23:49.833 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.721+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9801a4790 con 0x7ff980104c70 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.721+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9780115a0 con 0x7ff980104c70 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.721+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9801a4c90 con 0x7ff980104c70 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.722+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff98010fe10 con 0x7ff980104c70 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.723+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff978012a20 con 0x7ff980104c70 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff97dffb700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff9700a1260 0x7ff9700a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff984adc700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff9700a1260 0x7ff9700a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.834 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff9780cd2a0 con 0x7ff980104c70 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff97dffb700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] conn(0x7ff9700abdd0 0x7ff9700ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff9700ae8d0 con 0x7ff9700abdd0 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9780cd6b0 con 0x7ff980104c70 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.724+0000 7ff984adc700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff9700a1260 0x7ff9700a3710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff97400b5c0 tx=0x7ff974005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.725+0000 7ff97ffff700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] conn(0x7ff9700abdd0 0x7ff9700ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.725+0000 7ff97ffff700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] conn(0x7ff9700abdd0 0x7ff9700ae200 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.726+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== osd.11 v2:172.21.15.144:6824/852142335 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7ff9700ae8d0 con 0x7ff9700abdd0 2023-07-14T18:23:49.835 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.742+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 --> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff98004e340 con 0x7ff9700abdd0 2023-07-14T18:23:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.742+0000 7ff97dffb700 1 -- 172.21.15.64:0/2916957845 <== osd.11 v2:172.21.15.144:6824/852142335 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff98004e340 con 0x7ff9700abdd0 2023-07-14T18:23:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] conn(0x7ff9700abdd0 msgr2=0x7ff9700ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.836 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] conn(0x7ff9700abdd0 0x7ff9700ae200 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff9700a1260 msgr2=0x7ff9700a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff9700a1260 0x7ff9700a3710 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff97400b5c0 tx=0x7ff974005fb0 comp rx=0 tx=0).stop 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 msgr2=0x7ff9801a3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 --2- 172.21.15.64:0/2916957845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff980104c70 0x7ff9801a3e60 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff978007d70 tx=0x7ff97800a340 comp rx=0 tx=0).stop 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff9852dd700 1 -- 172.21.15.64:0/2916957845 reap_dead start 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 shutdown_connections 2023-07-14T18:23:49.837 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 >> 172.21.15.64:0/2916957845 conn(0x7ff980075280 msgr2=0x7ff9800fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:49.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 shutdown_connections 2023-07-14T18:23:49.838 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:49.743+0000 7ff986d40700 1 -- 172.21.15.64:0/2916957845 wait complete. 2023-07-14T18:23:50.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/2100575485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00ff7e0 msgr2=0x7fb3c01037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/2100575485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00ff7e0 0x7fb3c01037b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb3b40099c0 tx=0x7fb3b4009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/2100575485 shutdown_connections 2023-07-14T18:23:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/2100575485 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c00fee40 0x7fb3c00ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/2100575485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3c0103ee0 0x7fb3c010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/2100575485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00ff7e0 0x7fb3c01037b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/2100575485 >> 172.21.15.64:0/2100575485 conn(0x7fb3c0075ca0 msgr2=0x7fb3c00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:50.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/2100575485 shutdown_connections 2023-07-14T18:23:50.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.087+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/2100575485 wait complete. 2023-07-14T18:23:50.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.088+0000 7fb3c4dc1700 1 Processor -- start 2023-07-14T18:23:50.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.088+0000 7fb3c4dc1700 1 -- start start 2023-07-14T18:23:50.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 0x7fb3c019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c00ff7e0 0x7fb3c019aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3c0103ee0 0x7fb3c0194a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c0195070 con 0x7fb3c00fee40 2023-07-14T18:23:50.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01951e0 con 0x7fb3c0103ee0 2023-07-14T18:23:50.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3c4dc1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c01953a0 con 0x7fb3c00ff7e0 2023-07-14T18:23:50.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 0x7fb3c019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 0x7fb3c019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47654/0 (socket says 172.21.15.64:47654) 2023-07-14T18:23:50.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 -- 172.21.15.64:0/3847227987 learned_addr learned my addr 172.21.15.64:0/3847227987 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:50.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3bdd9b700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c00ff7e0 0x7fb3c019aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3bed9d700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3c0103ee0 0x7fb3c0194a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 -- 172.21.15.64:0/3847227987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c00ff7e0 msgr2=0x7fb3c019aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb3c00ff7e0 0x7fb3c019aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 -- 172.21.15.64:0/3847227987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3c0103ee0 msgr2=0x7fb3c0194a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3c0103ee0 0x7fb3c0194a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b40095f0 con 0x7fb3c00fee40 2023-07-14T18:23:50.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3be59c700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 0x7fb3c019a980 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000e9a0 tx=0x7fb3b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.089+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0009790 con 0x7fb3c00fee40 2023-07-14T18:23:50.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.090+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb3b0010430 con 0x7fb3c00fee40 2023-07-14T18:23:50.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.090+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c0195680 con 0x7fb3c00fee40 2023-07-14T18:23:50.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.090+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3b000cea0 con 0x7fb3c00fee40 2023-07-14T18:23:50.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.090+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c01a1970 con 0x7fb3c00fee40 2023-07-14T18:23:50.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.090+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb3c004e340 con 0x7fb3c00fee40 2023-07-14T18:23:50.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.092+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb3b00105a0 con 0x7fb3c00fee40 2023-07-14T18:23:50.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.092+0000 7fb3af7fe700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb3a80a11e0 0x7fb3a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.092+0000 7fb3bdd9b700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb3a80a11e0 0x7fb3a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb3b0014070 con 0x7fb3c00fee40 2023-07-14T18:23:50.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3af7fe700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] conn(0x7fb3a80abd50 0x7fb3a80ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb3a80ae850 con 0x7fb3a80abd50 2023-07-14T18:23:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3b00cd500 con 0x7fb3c00fee40 2023-07-14T18:23:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3bed9d700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] conn(0x7fb3a80abd50 0x7fb3a80ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3bdd9b700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb3a80a11e0 0x7fb3a80a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009990 tx=0x7fb3b4005be0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.093+0000 7fb3bed9d700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] conn(0x7fb3a80abd50 0x7fb3a80ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.094+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== osd.3 v2:172.21.15.64:6810/2893239811 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb3a80ae850 con 0x7fb3a80abd50 2023-07-14T18:23:50.235 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.112+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 --> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb3c00fce40 con 0x7fb3a80abd50 2023-07-14T18:23:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.112+0000 7fb3af7fe700 1 -- 172.21.15.64:0/3847227987 <== osd.3 v2:172.21.15.64:6810/2893239811 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb3c00fce40 con 0x7fb3a80abd50 2023-07-14T18:23:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.112+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] conn(0x7fb3a80abd50 msgr2=0x7fb3a80ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.112+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] conn(0x7fb3a80abd50 0x7fb3a80ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.236 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.112+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb3a80a11e0 msgr2=0x7fb3a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb3a80a11e0 0x7fb3a80a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3b4009990 tx=0x7fb3b4005be0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 msgr2=0x7fb3c019a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.237 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 --2- 172.21.15.64:0/3847227987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3c00fee40 0x7fb3c019a980 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb3b000e9a0 tx=0x7fb3b000ecb0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3bed9d700 1 -- 172.21.15.64:0/3847227987 reap_dead start 2023-07-14T18:23:50.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 shutdown_connections 2023-07-14T18:23:50.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 >> 172.21.15.64:0/3847227987 conn(0x7fb3c0075ca0 msgr2=0x7fb3c00fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:50.238 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 shutdown_connections 2023-07-14T18:23:50.239 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.113+0000 7fb3c4dc1700 1 -- 172.21.15.64:0/3847227987 wait complete. 2023-07-14T18:23:50.585 INFO:teuthology.orchestra.run.smithi064.stdout:141733920818 2023-07-14T18:23:50.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T18:23:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 -- 172.21.15.64:0/483061488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374072980 msgr2=0x7f1374072d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 --2- 172.21.15.64:0/483061488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374072980 0x7f1374072d50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1364009990 tx=0x7f1364009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 -- 172.21.15.64:0/483061488 shutdown_connections 2023-07-14T18:23:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 --2- 172.21.15.64:0/483061488 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f137410c570 0x7f137410e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 --2- 172.21.15.64:0/483061488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374072980 0x7f1374072d50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 --2- 172.21.15.64:0/483061488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1374073290 0x7f137410c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.584+0000 7f137a945700 1 -- 172.21.15.64:0/483061488 >> 172.21.15.64:0/483061488 conn(0x7f13740fb790 msgr2=0x7f13740fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.585+0000 7f137a945700 1 -- 172.21.15.64:0/483061488 shutdown_connections 2023-07-14T18:23:50.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.585+0000 7f137a945700 1 -- 172.21.15.64:0/483061488 wait complete. 2023-07-14T18:23:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.585+0000 7f137a945700 1 Processor -- start 2023-07-14T18:23:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.585+0000 7f137a945700 1 -- start start 2023-07-14T18:23:50.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1374072980 0x7f1374197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 0x7f1374198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f137410c570 0x7f137419c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419ceb0 con 0x7f1374073290 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419d020 con 0x7f137410c570 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f137a945700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137419d1e0 con 0x7f1374072980 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 0x7f1374198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 0x7f1374198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47672/0 (socket says 172.21.15.64:47672) 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 -- 172.21.15.64:0/1651504125 learned_addr learned my addr 172.21.15.64:0/1651504125 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f1378ee2700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f137410c570 0x7f137419c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f1373fff700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1374072980 0x7f1374197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 -- 172.21.15.64:0/1651504125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1374072980 msgr2=0x7f1374197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1374072980 0x7f1374197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 -- 172.21.15.64:0/1651504125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f137410c570 msgr2=0x7f137419c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f137410c570 0x7f137419c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13640095f0 con 0x7f1374073290 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f1373fff700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1374072980 0x7f1374197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.586+0000 7f13737fe700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 0x7f1374198430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f136800d950 tx=0x7f136800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.587+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1368009940 con 0x7f1374073290 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.587+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1368010460 con 0x7f1374073290 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.587+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137419d4c0 con 0x7f1374073290 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.587+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800f620 con 0x7f1374073290 2023-07-14T18:23:50.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.587+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13741a4d20 con 0x7f1374073290 2023-07-14T18:23:50.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.588+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1374111f40 con 0x7f1374073290 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.589+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f13680105d0 con 0x7f1374073290 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f13717fa700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f135c0a11e0 0x7f135c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f13680cd010 con 0x7f1374073290 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f1373fff700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f135c0a11e0 0x7f135c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f13717fa700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] conn(0x7f135c0abd50 0x7f135c0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:50.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f135c0ae850 con 0x7f135c0abd50 2023-07-14T18:23:50.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13680cd420 con 0x7f1374073290 2023-07-14T18:23:50.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f1378ee2700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] conn(0x7f135c0abd50 0x7f135c0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:50.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.590+0000 7f1373fff700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f135c0a11e0 0x7f135c0a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f136400b5c0 tx=0x7f1364005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.591+0000 7f1378ee2700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] conn(0x7f135c0abd50 0x7f135c0ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:50.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.591+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== osd.0 v2:172.21.15.64:6802/1693019669 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f135c0ae850 con 0x7f135c0abd50 2023-07-14T18:23:50.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.608+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 --> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f137404e340 con 0x7f135c0abd50 2023-07-14T18:23:50.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.608+0000 7f13717fa700 1 -- 172.21.15.64:0/1651504125 <== osd.0 v2:172.21.15.64:6802/1693019669 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f137404e340 con 0x7f135c0abd50 2023-07-14T18:23:50.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] conn(0x7f135c0abd50 msgr2=0x7f135c0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] conn(0x7f135c0abd50 0x7f135c0ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f135c0a11e0 msgr2=0x7f135c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f135c0a11e0 0x7f135c0a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f136400b5c0 tx=0x7f1364005fd0 comp rx=0 tx=0).stop 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 msgr2=0x7f1374198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 --2- 172.21.15.64:0/1651504125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1374073290 0x7f1374198430 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f136800d950 tx=0x7f136800dc60 comp rx=0 tx=0).stop 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f1378ee2700 1 -- 172.21.15.64:0/1651504125 reap_dead start 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 shutdown_connections 2023-07-14T18:23:50.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 >> 172.21.15.64:0/1651504125 conn(0x7f13740fb790 msgr2=0x7f137410e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:50.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 shutdown_connections 2023-07-14T18:23:50.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:50.609+0000 7f137a945700 1 -- 172.21.15.64:0/1651504125 wait complete. 2023-07-14T18:23:51.081 INFO:teuthology.orchestra.run.smithi064.stdout:107374182454 2023-07-14T18:23:51.082 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T18:23:51.528 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:51 smithi064 ceph-mon[96720]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:51.530 INFO:teuthology.orchestra.run.smithi064.stdout:244813135905 2023-07-14T18:23:51.530 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T18:23:51.531 INFO:teuthology.orchestra.run.smithi064.stdout:326417514516 2023-07-14T18:23:51.532 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T18:23:51.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:51 smithi144 ceph-mon[98417]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:51.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:51 smithi105 ceph-mon[98013]: pgmap v277: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:52.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.095+0000 7fb7fe198700 1 -- 172.21.15.64:0/4159776411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f8103f10 msgr2=0x7fb7f810ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.095+0000 7fb7fe198700 1 --2- 172.21.15.64:0/4159776411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f8103f10 0x7fb7f810ba90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000b0d0 tx=0x7fb7f000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 -- 172.21.15.64:0/4159776411 shutdown_connections 2023-07-14T18:23:52.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 --2- 172.21.15.64:0/4159776411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f80fee70 0x7fb7f80ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 --2- 172.21.15.64:0/4159776411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f8103f10 0x7fb7f810ba90 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 --2- 172.21.15.64:0/4159776411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7f80ff810 0x7fb7f81037e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 -- 172.21.15.64:0/4159776411 >> 172.21.15.64:0/4159776411 conn(0x7fb7f8075d20 msgr2=0x7fb7f8076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:52.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 -- 172.21.15.64:0/4159776411 shutdown_connections 2023-07-14T18:23:52.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 -- 172.21.15.64:0/4159776411 wait complete. 2023-07-14T18:23:52.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.096+0000 7fb7fe198700 1 Processor -- start 2023-07-14T18:23:52.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 -- start start 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 0x7fb7f8194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7f80ff810 0x7fb7f81952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f8103f10 0x7fb7f8199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8199d30 con 0x7fb7f80fee70 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f8199ea0 con 0x7fb7f80ff810 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7fe198700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f819a060 con 0x7fb7f8103f10 2023-07-14T18:23:52.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 0x7fb7f8194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 0x7fb7f8194d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47684/0 (socket says 172.21.15.64:47684) 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 -- 172.21.15.64:0/2584000091 learned_addr learned my addr 172.21.15.64:0/2584000091 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f7fff700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f8103f10 0x7fb7f8199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f6ffd700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7f80ff810 0x7fb7f81952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 -- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f8103f10 msgr2=0x7fb7f8199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f8103f10 0x7fb7f8199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 -- 172.21.15.64:0/2584000091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7f80ff810 msgr2=0x7fb7f81952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb7f80ff810 0x7fb7f81952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f0009d00 con 0x7fb7f80fee70 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f7fff700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb7f8103f10 0x7fb7f8199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.097+0000 7fb7f77fe700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 0x7fb7f8194d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009970 tx=0x7fb7e8009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.098+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8012070 con 0x7fb7f80fee70 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.098+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f819a340 con 0x7fb7f80fee70 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.098+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb7e800d260 con 0x7fb7f80fee70 2023-07-14T18:23:52.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.098+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7e8017900 con 0x7fb7f80fee70 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.098+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f8068bf0 con 0x7fb7f80fee70 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.099+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb7e4000ff0 con 0x7fb7f80fee70 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.100+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb7e8017a60 con 0x7fb7f80fee70 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.100+0000 7fb7f4ff9700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7e00a1260 0x7fb7e00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f6ffd700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7e00a1260 0x7fb7e00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb7e80cdf90 con 0x7fb7f80fee70 2023-07-14T18:23:52.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f4ff9700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] conn(0x7fb7e00abdd0 0x7fb7e00ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb7e00ae8d0 con 0x7fb7e00abdd0 2023-07-14T18:23:52.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7e80d0610 con 0x7fb7f80fee70 2023-07-14T18:23:52.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f6ffd700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7e00a1260 0x7fb7e00a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec005950 tx=0x7fb7ec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f7fff700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] conn(0x7fb7e00abdd0 0x7fb7e00ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.101+0000 7fb7f7fff700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] conn(0x7fb7e00abdd0 0x7fb7e00ae200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.102+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== osd.2 v2:172.21.15.144:6800/1093978507 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb7e00ae8d0 con 0x7fb7e00abdd0 2023-07-14T18:23:52.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.122+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 --> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb7e4002d70 con 0x7fb7e00abdd0 2023-07-14T18:23:52.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.123+0000 7fb7f4ff9700 1 -- 172.21.15.64:0/2584000091 <== osd.2 v2:172.21.15.144:6800/1093978507 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb7e4002d70 con 0x7fb7e00abdd0 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.123+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] conn(0x7fb7e00abdd0 msgr2=0x7fb7e00ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.123+0000 7fb7fe198700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] conn(0x7fb7e00abdd0 0x7fb7e00ae200 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7e00a1260 msgr2=0x7fb7e00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb7e00a1260 0x7fb7e00a3710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec005950 tx=0x7fb7ec0058e0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 msgr2=0x7fb7f8194d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 --2- 172.21.15.64:0/2584000091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7f80fee70 0x7fb7f8194d70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb7e8009970 tx=0x7fb7e8009d30 comp rx=0 tx=0).stop 2023-07-14T18:23:52.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7f7fff700 1 -- 172.21.15.64:0/2584000091 reap_dead start 2023-07-14T18:23:52.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 shutdown_connections 2023-07-14T18:23:52.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 >> 172.21.15.64:0/2584000091 conn(0x7fb7f8075d20 msgr2=0x7fb7f80fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:52.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 shutdown_connections 2023-07-14T18:23:52.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.124+0000 7fb7fe198700 1 -- 172.21.15.64:0/2584000091 wait complete. 2023-07-14T18:23:52.257 INFO:teuthology.orchestra.run.smithi064.stdout:34359738436 2023-07-14T18:23:52.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T18:23:52.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 -- 172.21.15.64:0/3568705361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 msgr2=0x7efe5410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/3568705361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe5410ba60 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00b0d0 tx=0x7efe4c00b3e0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 -- 172.21.15.64:0/3568705361 shutdown_connections 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/3568705361 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe540fee40 0x7efe540ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/3568705361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe5410ba60 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/3568705361 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe540ff7e0 0x7efe541037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 -- 172.21.15.64:0/3568705361 >> 172.21.15.64:0/3568705361 conn(0x7efe54075ca0 msgr2=0x7efe540760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:52.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 -- 172.21.15.64:0/3568705361 shutdown_connections 2023-07-14T18:23:52.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.338+0000 7efe5c7cf700 1 -- 172.21.15.64:0/3568705361 wait complete. 2023-07-14T18:23:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 Processor -- start 2023-07-14T18:23:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 -- start start 2023-07-14T18:23:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe540fee40 0x7efe5419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe540ff7e0 0x7efe5419b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe54194b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe541951c0 con 0x7efe54103ee0 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe54195330 con 0x7efe540ff7e0 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.339+0000 7efe5c7cf700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe541954f0 con 0x7efe540fee40 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe54194b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe54194b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47704/0 (socket says 172.21.15.64:47704) 2023-07-14T18:23:52.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 -- 172.21.15.64:0/1682483716 learned_addr learned my addr 172.21.15.64:0/1682483716 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5a56b700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe540fee40 0x7efe5419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe59d6a700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe540ff7e0 0x7efe5419b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 -- 172.21.15.64:0/1682483716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe540fee40 msgr2=0x7efe5419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efe540fee40 0x7efe5419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 -- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe540ff7e0 msgr2=0x7efe5419b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe540ff7e0 0x7efe5419b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c009d00 con 0x7efe54103ee0 2023-07-14T18:23:52.394 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5ad6c700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe54194b50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efe4c008e50 tx=0x7efe4c008e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c00e040 con 0x7efe54103ee0 2023-07-14T18:23:52.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe54195770 con 0x7efe54103ee0 2023-07-14T18:23:52.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efe4c003e90 con 0x7efe54103ee0 2023-07-14T18:23:52.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.340+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c01b460 con 0x7efe54103ee0 2023-07-14T18:23:52.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.341+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe540682d0 con 0x7efe54103ee0 2023-07-14T18:23:52.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.343+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efe4c003a00 con 0x7efe54103ee0 2023-07-14T18:23:52.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.343+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efe5410ebc0 con 0x7efe54103ee0 2023-07-14T18:23:52.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.343+0000 7efe4b7fe700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efe400a11e0 0x7efe400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efe4c0cdc80 con 0x7efe54103ee0 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe5a56b700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efe400a11e0 0x7efe400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe4b7fe700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] conn(0x7efe400abd50 0x7efe400ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efe400ae850 con 0x7efe400abd50 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe4c0d0300 con 0x7efe54103ee0 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe5a56b700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efe400a11e0 0x7efe400a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efe50007be0 tx=0x7efe500073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe59d6a700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] conn(0x7efe400abd50 0x7efe400ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:52.398 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.344+0000 7efe59d6a700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] conn(0x7efe400abd50 0x7efe400ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:52.398 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.346+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== osd.10 v2:172.21.15.105:6824/1056583141 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7efe400ae850 con 0x7efe400abd50 2023-07-14T18:23:52.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.364+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 --> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efe5404e340 con 0x7efe400abd50 2023-07-14T18:23:52.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe4b7fe700 1 -- 172.21.15.64:0/1682483716 <== osd.10 v2:172.21.15.105:6824/1056583141 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efe5404e340 con 0x7efe400abd50 2023-07-14T18:23:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] conn(0x7efe400abd50 msgr2=0x7efe400ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] conn(0x7efe400abd50 0x7efe400ae180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.418 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efe400a11e0 msgr2=0x7efe400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efe400a11e0 0x7efe400a3690 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efe50007be0 tx=0x7efe500073d0 comp rx=0 tx=0).stop 2023-07-14T18:23:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 msgr2=0x7efe54194b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:52.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5c7cf700 1 --2- 172.21.15.64:0/1682483716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe54103ee0 0x7efe54194b50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7efe4c008e50 tx=0x7efe4c008e80 comp rx=0 tx=0).stop 2023-07-14T18:23:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.365+0000 7efe5ad6c700 1 -- 172.21.15.64:0/1682483716 reap_dead start 2023-07-14T18:23:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.366+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 shutdown_connections 2023-07-14T18:23:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.366+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 >> 172.21.15.64:0/1682483716 conn(0x7efe54075ca0 msgr2=0x7efe540fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:52.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.366+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 shutdown_connections 2023-07-14T18:23:52.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:52.366+0000 7efe5c7cf700 1 -- 172.21.15.64:0/1682483716 wait complete. 2023-07-14T18:23:52.534 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:52.742 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.022 INFO:teuthology.orchestra.run.smithi064.stdout:296352743449 2023-07-14T18:23:53.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T18:23:53.026 INFO:teuthology.orchestra.run.smithi064.stdout:77309411385 2023-07-14T18:23:53.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T18:23:53.207 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.208 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.438 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.471 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.472 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.611 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:53.666 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:53 smithi064 ceph-mon[96720]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T18:23:53.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:53 smithi144 ceph-mon[98417]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T18:23:53.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:53 smithi105 ceph-mon[98013]: pgmap v278: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2023-07-14T18:23:54.074 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:55.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:55 smithi144 ceph-mon[98417]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:55.786 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:55 smithi064 ceph-mon[96720]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:55.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:55 smithi105 ceph-mon[98013]: pgmap v279: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2023-07-14T18:23:56.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 -- 172.21.15.64:0/1582526860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0100980 msgr2=0x7f16e01090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 --2- 172.21.15.64:0/1582526860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0100980 0x7f16e01090e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009990 tx=0x7f16c8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 -- 172.21.15.64:0/1582526860 shutdown_connections 2023-07-14T18:23:56.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 --2- 172.21.15.64:0/1582526860 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0109620 0x7f16e010ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 --2- 172.21.15.64:0/1582526860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 0x7f16e0100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 --2- 172.21.15.64:0/1582526860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0100980 0x7f16e01090e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 -- 172.21.15.64:0/1582526860 >> 172.21.15.64:0/1582526860 conn(0x7f16e0074550 msgr2=0x7f16e0074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 -- 172.21.15.64:0/1582526860 shutdown_connections 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.656+0000 7f16e5317700 1 -- 172.21.15.64:0/1582526860 wait complete. 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 Processor -- start 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 -- start start 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 0x7f16e0195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0100980 0x7f16e0196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e0199bd0 con 0x7f16e0109620 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e0199d40 con 0x7f16e0100070 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16e5317700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e0199f00 con 0x7f16e0100980 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47720/0 (socket says 172.21.15.64:47720) 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 -- 172.21.15.64:0/3577231619 learned_addr learned my addr 172.21.15.64:0/3577231619 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:56.711 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16deffd700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 0x7f16e0195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16d7fff700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0100980 0x7f16e0196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 -- 172.21.15.64:0/3577231619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0100980 msgr2=0x7f16e0196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0100980 0x7f16e0196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 -- 172.21.15.64:0/3577231619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 msgr2=0x7f16e0195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.657+0000 7f16df7fe700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 0x7f16e0195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16df7fe700 1 -- 172.21.15.64:0/3577231619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c80095f0 con 0x7f16e0109620 2023-07-14T18:23:56.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16df7fe700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009fd0 tx=0x7f16d800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:56.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d8018070 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f16d800f040 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16d800ccc0 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e019a180 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.658+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e01a2b30 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.659+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e004e340 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.660+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f16d8003a00 con 0x7f16e0109620 2023-07-14T18:23:56.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.661+0000 7f16dcff9700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 0x7f16cc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.661+0000 7f16deffd700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 0x7f16cc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.661+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f16d80cc7e0 con 0x7f16e0109620 2023-07-14T18:23:56.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.661+0000 7f16deffd700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 0x7f16cc0a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16d0005950 tx=0x7f16d000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:56.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.665+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f16d8095c10 con 0x7f16e0109620 2023-07-14T18:23:56.900 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.846+0000 7f08c7773700 1 -- 172.21.15.64:0/3662273503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c0100f20 msgr2=0x7f08c01012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.900 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.846+0000 7f08c7773700 1 --2- 172.21.15.64:0/3662273503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c0100f20 0x7f08c01012f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f08bc009990 tx=0x7f08bc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.846+0000 7f08c7773700 1 -- 172.21.15.64:0/3662273503 shutdown_connections 2023-07-14T18:23:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 --2- 172.21.15.64:0/3662273503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c010a270 0x7f08c010c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 --2- 172.21.15.64:0/3662273503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c0100f20 0x7f08c01012f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 --2- 172.21.15.64:0/3662273503 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0101830 0x7f08c0109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 -- 172.21.15.64:0/3662273503 >> 172.21.15.64:0/3662273503 conn(0x7f08c0075ca0 msgr2=0x7f08c00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 -- 172.21.15.64:0/3662273503 shutdown_connections 2023-07-14T18:23:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 -- 172.21.15.64:0/3662273503 wait complete. 2023-07-14T18:23:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 Processor -- start 2023-07-14T18:23:56.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.847+0000 7f08c7773700 1 -- start start 2023-07-14T18:23:56.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0100f20 0x7f08c019ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c0101830 0x7f08c019bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0196100 con 0x7f08c010a270 2023-07-14T18:23:56.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0196270 con 0x7f08c0101830 2023-07-14T18:23:56.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c7773700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08c0196430 con 0x7f08c0100f20 2023-07-14T18:23:56.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.904 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47734/0 (socket says 172.21.15.64:47734) 2023-07-14T18:23:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 -- 172.21.15.64:0/382738933 learned_addr learned my addr 172.21.15.64:0/382738933 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c4d0e700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c0101830 0x7f08c019bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c550f700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0100f20 0x7f08c019ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.905 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 -- 172.21.15.64:0/382738933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0100f20 msgr2=0x7f08c019ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0100f20 0x7f08c019ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 -- 172.21.15.64:0/382738933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c0101830 msgr2=0x7f08c019bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c0101830 0x7f08c019bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 -- 172.21.15.64:0/382738933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc0095f0 con 0x7f08c010a270 2023-07-14T18:23:56.906 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.848+0000 7f08c5d10700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f08b800cca0 tx=0x7f08b800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:56.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.849+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b800aaf0 con 0x7f08c010a270 2023-07-14T18:23:56.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.849+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f08b800ac50 con 0x7f08c010a270 2023-07-14T18:23:56.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.849+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08b80115e0 con 0x7f08c010a270 2023-07-14T18:23:56.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.849+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08c0196710 con 0x7f08c010a270 2023-07-14T18:23:56.907 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.849+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08c00fec80 con 0x7f08c010a270 2023-07-14T18:23:56.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.850+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08c0104b60 con 0x7f08c010a270 2023-07-14T18:23:56.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.851+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f08b8004ad0 con 0x7f08c010a270 2023-07-14T18:23:56.908 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.852+0000 7f08b67fc700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 0x7f08ac0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:56.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.852+0000 7f08c550f700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 0x7f08ac0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:56.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.852+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f08b80cd090 con 0x7f08c010a270 2023-07-14T18:23:56.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.852+0000 7f08c550f700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 0x7f08ac0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00b5c0 tx=0x7f08bc009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:56.909 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.855+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f08b80965a0 con 0x7f08c010a270 2023-07-14T18:23:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.874+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f16e01a2d50 con 0x7f16e0109620 2023-07-14T18:23:56.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.874+0000 7f16dcff9700 1 -- 172.21.15.64:0/3577231619 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f16d8099230 con 0x7f16e0109620 2023-07-14T18:23:56.926 INFO:teuthology.orchestra.run.smithi064.stdout:163208757295 2023-07-14T18:23:56.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 msgr2=0x7f16cc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 0x7f16cc0a3710 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f16d0005950 tx=0x7f16d000b410 comp rx=0 tx=0).stop 2023-07-14T18:23:56.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 msgr2=0x7f16e01995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f16d8009fd0 tx=0x7f16d800ef90 comp rx=0 tx=0).stop 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 shutdown_connections 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f16cc0a1260 0x7f16cc0a3710 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16e0109620 0x7f16e01995d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f16e0100070 0x7f16e0195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 --2- 172.21.15.64:0/3577231619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f16e0100980 0x7f16e0196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 >> 172.21.15.64:0/3577231619 conn(0x7f16e0074550 msgr2=0x7f16e00fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:56.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 shutdown_connections 2023-07-14T18:23:56.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:56.877+0000 7f16e5317700 1 -- 172.21.15.64:0/3577231619 wait complete. 2023-07-14T18:23:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.075+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f08c004e340 con 0x7f08c010a270 2023-07-14T18:23:57.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.075+0000 7f08b67fc700 1 -- 172.21.15.64:0/382738933 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f08b80cae60 con 0x7f08c010a270 2023-07-14T18:23:57.127 INFO:teuthology.orchestra.run.smithi064.stdout:193273528363 2023-07-14T18:23:57.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 msgr2=0x7f08ac0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 0x7f08ac0a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f08bc00b5c0 tx=0x7f08bc009f00 comp rx=0 tx=0).stop 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 msgr2=0x7f08c0195a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f08b800cca0 tx=0x7f08b800a060 comp rx=0 tx=0).stop 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 shutdown_connections 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f08c0100f20 0x7f08c019ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f08ac0a1260 0x7f08ac0a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:57.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08c010a270 0x7f08c0195a90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 --2- 172.21.15.64:0/382738933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f08c0101830 0x7f08c019bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 >> 172.21.15.64:0/382738933 conn(0x7f08c0075ca0 msgr2=0x7f08c0101ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 shutdown_connections 2023-07-14T18:23:57.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:57.077+0000 7f08c7773700 1 -- 172.21.15.64:0/382738933 wait complete. 2023-07-14T18:23:57.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:57 smithi064 ceph-mon[96720]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:23:57.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:57 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3577231619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:23:57.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:57 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/382738933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:23:57.661 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:57.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:57 smithi144 ceph-mon[98417]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:23:57.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:57 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3577231619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:23:57.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:57 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/382738933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:23:57.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:57 smithi105 ceph-mon[98013]: pgmap v280: 129 pgs: 129 active+clean; 639 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:23:57.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:57 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3577231619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T18:23:57.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:57 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/382738933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T18:23:58.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 -- 172.21.15.64:0/3193559757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80ff7e0 msgr2=0x7f36d81037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:58.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/3193559757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80ff7e0 0x7f36d81037b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f36cc0099c0 tx=0x7f36cc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 -- 172.21.15.64:0/3193559757 shutdown_connections 2023-07-14T18:23:58.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/3193559757 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d8103ee0 0x7f36d810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/3193559757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d80fee40 0x7f36d80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/3193559757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80ff7e0 0x7f36d81037b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 -- 172.21.15.64:0/3193559757 >> 172.21.15.64:0/3193559757 conn(0x7f36d8075ca0 msgr2=0x7f36d80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:58.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 -- 172.21.15.64:0/3193559757 shutdown_connections 2023-07-14T18:23:58.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.022+0000 7f36dd6e3700 1 -- 172.21.15.64:0/3193559757 wait complete. 2023-07-14T18:23:58.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.023+0000 7f36dd6e3700 1 Processor -- start 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.023+0000 7f36dd6e3700 1 -- start start 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.023+0000 7f36dd6e3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.023+0000 7f36dd6e3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 0x7f36d81970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d8103ee0 0x7f36d81a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81979c0 con 0x7f36d80fee40 2023-07-14T18:23:58.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81a2130 con 0x7f36d8103ee0 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81a22a0 con 0x7f36d80ff7e0 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d67fc700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 0x7f36d81970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47742/0 (socket says 172.21.15.64:47742) 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 -- 172.21.15.64:0/1772572681 learned_addr learned my addr 172.21.15.64:0/1772572681 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d67fc700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 0x7f36d81970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:55400/0 (socket says 172.21.15.64:55400) 2023-07-14T18:23:58.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d77fe700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d8103ee0 0x7f36d81a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 -- 172.21.15.64:0/1772572681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 msgr2=0x7f36d81970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 0x7f36d81970b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 -- 172.21.15.64:0/1772572681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d8103ee0 msgr2=0x7f36d81a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d8103ee0 0x7f36d81a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 -- 172.21.15.64:0/1772572681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc0095f0 con 0x7f36d80fee40 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36d6ffd700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c920 tx=0x7f36c800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8004a30 con 0x7f36d80fee40 2023-07-14T18:23:58.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d81949c0 con 0x7f36d80fee40 2023-07-14T18:23:58.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f36c80054b0 con 0x7f36d80fee40 2023-07-14T18:23:58.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c8007ab0 con 0x7f36d80fee40 2023-07-14T18:23:58.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.024+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d8194e90 con 0x7f36d80fee40 2023-07-14T18:23:58.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.025+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d804e340 con 0x7f36d80fee40 2023-07-14T18:23:58.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.026+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f36c8007c30 con 0x7f36d80fee40 2023-07-14T18:23:58.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.027+0000 7f36bffff700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 0x7f36c00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:58.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.027+0000 7f36d67fc700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 0x7f36c00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:58.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.027+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f36c80ccc30 con 0x7f36d80fee40 2023-07-14T18:23:58.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.028+0000 7f36d67fc700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 0x7f36c00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36cc006010 tx=0x7f36cc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:58.086 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.030+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f36c8096030 con 0x7f36d80fee40 2023-07-14T18:23:58.283 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.232+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f36d81951f0 con 0x7f36d80fee40 2023-07-14T18:23:58.284 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.232+0000 7f36bffff700 1 -- 172.21.15.64:0/1772572681 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f36c801d020 con 0x7f36d80fee40 2023-07-14T18:23:58.284 INFO:teuthology.orchestra.run.smithi064.stdout:326417514517 2023-07-14T18:23:58.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 msgr2=0x7f36c00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:58.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 0x7f36c00a3690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f36cc006010 tx=0x7f36cc009f10 comp rx=0 tx=0).stop 2023-07-14T18:23:58.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 msgr2=0x7f36d8196b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:58.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f36c800c920 tx=0x7f36c800cc30 comp rx=0 tx=0).stop 2023-07-14T18:23:58.286 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 shutdown_connections 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f36c00a11e0 0x7f36c00a3690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f36d8103ee0 0x7f36d81a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36d80fee40 0x7f36d8196b70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 --2- 172.21.15.64:0/1772572681 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f36d80ff7e0 0x7f36d81970b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 >> 172.21.15.64:0/1772572681 conn(0x7f36d8075ca0 msgr2=0x7f36d80fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:58.287 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 shutdown_connections 2023-07-14T18:23:58.288 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:58.235+0000 7f36dd6e3700 1 -- 172.21.15.64:0/1772572681 wait complete. 2023-07-14T18:23:58.464 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528360 got 193273528363 for osd.6 2023-07-14T18:23:58.464 DEBUG:teuthology.parallel:result is None 2023-07-14T18:23:58.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:58 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1772572681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:23:58.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:58 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1772572681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:23:58.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:58 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1772572681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T18:23:58.932 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757293 got 163208757295 for osd.5 2023-07-14T18:23:58.932 DEBUG:teuthology.parallel:result is None 2023-07-14T18:23:59.425 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:23:59 smithi064 ceph-mon[96720]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:23:59.665 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:23:59.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:23:59 smithi144 ceph-mon[98417]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:23:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1043047271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 msgr2=0x7f0ad4100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1043047271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad4100400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8009990 tx=0x7f0ac8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1043047271 shutdown_connections 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1043047271 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad41095e0 0x7f0ad410b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1043047271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad4100400 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1043047271 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad4100940 0x7f0ad41090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1043047271 >> 172.21.15.64:0/1043047271 conn(0x7f0ad4075480 msgr2=0x7f0ad4075880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1043047271 shutdown_connections 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1043047271 wait complete. 2023-07-14T18:23:59.729 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 Processor -- start 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.676+0000 7f0adb1ce700 1 -- start start 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad4100940 0x7f0ad419c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 0x7f0ad4195b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41961b0 con 0x7f0ad4100030 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad4196320 con 0x7f0ad4100940 2023-07-14T18:23:59.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0adb1ce700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad41964e0 con 0x7f0ad41095e0 2023-07-14T18:23:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47756/0 (socket says 172.21.15.64:47756) 2023-07-14T18:23:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 -- 172.21.15.64:0/1005360022 learned_addr learned my addr 172.21.15.64:0/1005360022 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:23:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad3fff700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad4100940 0x7f0ad419c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:59.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad976b700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 0x7f0ad4195b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:59.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 -- 172.21.15.64:0/1005360022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 msgr2=0x7f0ad4195b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 0x7f0ad4195b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 -- 172.21.15.64:0/1005360022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad4100940 msgr2=0x7f0ad419c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad4100940 0x7f0ad419c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 -- 172.21.15.64:0/1005360022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80095f0 con 0x7f0ad4100030 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad976b700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 0x7f0ad4195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.677+0000 7f0ad8f6a700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8006010 tx=0x7f0ac8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.678+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac801d070 con 0x7f0ad4100030 2023-07-14T18:23:59.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.678+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ac8004000 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.678+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8022c00 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.678+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad4196760 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.678+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad41a2ab0 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.679+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad404e340 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.680+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0ac8022470 con 0x7f0ad4100030 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.680+0000 7f0ad1ffb700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 0x7f0ac40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:23:59.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.681+0000 7f0ad3fff700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 0x7f0ac40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:23:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.681+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0ac80ce270 con 0x7f0ad4100030 2023-07-14T18:23:59.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.681+0000 7f0ad3fff700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 0x7f0ac40a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40ffe60 tx=0x7f0ac00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:23:59.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.685+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ac80977c0 con 0x7f0ad4100030 2023-07-14T18:23:59.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:23:59 smithi105 ceph-mon[98013]: pgmap v281: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:23:59.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.912+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f0ad4196fe0 con 0x7f0ad4100030 2023-07-14T18:23:59.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.913+0000 7f0ad1ffb700 1 -- 172.21.15.64:0/1005360022 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f0ac8027020 con 0x7f0ad4100030 2023-07-14T18:23:59.965 INFO:teuthology.orchestra.run.smithi064.stdout:141733920820 2023-07-14T18:23:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.915+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 msgr2=0x7f0ac40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:59.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.915+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 0x7f0ac40a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40ffe60 tx=0x7f0ac00073d0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.915+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 msgr2=0x7f0ad419bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.915+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0ac8006010 tx=0x7f0ac8005740 comp rx=0 tx=0).stop 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 shutdown_connections 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0ad41095e0 0x7f0ad4195b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f0ac40a11e0 0x7f0ac40a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0ad4100030 0x7f0ad419bac0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 --2- 172.21.15.64:0/1005360022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0ad4100940 0x7f0ad419c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:23:59.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 >> 172.21.15.64:0/1005360022 conn(0x7f0ad4075480 msgr2=0x7f0ad40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:23:59.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 shutdown_connections 2023-07-14T18:23:59.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:23:59.916+0000 7f0adb1ce700 1 -- 172.21.15.64:0/1005360022 wait complete. 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 -- 172.21.15.64:0/1928961260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240ff7e0 msgr2=0x7f32241037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 --2- 172.21.15.64:0/1928961260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240ff7e0 0x7f32241037b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f32180099c0 tx=0x7f3218009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 -- 172.21.15.64:0/1928961260 shutdown_connections 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 --2- 172.21.15.64:0/1928961260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32240fee40 0x7f32240ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 --2- 172.21.15.64:0/1928961260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3224103ee0 0x7f322410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 --2- 172.21.15.64:0/1928961260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240ff7e0 0x7f32241037b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.089+0000 7f322c3b4700 1 -- 172.21.15.64:0/1928961260 >> 172.21.15.64:0/1928961260 conn(0x7f3224075ca0 msgr2=0x7f32240760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:00.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.091+0000 7f322c3b4700 1 -- 172.21.15.64:0/1928961260 shutdown_connections 2023-07-14T18:24:00.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.091+0000 7f322c3b4700 1 -- 172.21.15.64:0/1928961260 wait complete. 2023-07-14T18:24:00.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.091+0000 7f322c3b4700 1 Processor -- start 2023-07-14T18:24:00.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.091+0000 7f322c3b4700 1 -- start start 2023-07-14T18:24:00.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32240ff7e0 0x7f3224194f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224103ee0 0x7f32241954c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32241a4280 con 0x7f32240fee40 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32241a43f0 con 0x7f32240ff7e0 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322c3b4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32241a45b0 con 0x7f3224103ee0 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47768/0 (socket says 172.21.15.64:47768) 2023-07-14T18:24:00.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 -- 172.21.15.64:0/2657778817 learned_addr learned my addr 172.21.15.64:0/2657778817 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322994f700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32240ff7e0 0x7f3224194f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a951700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224103ee0 0x7f32241954c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 -- 172.21.15.64:0/2657778817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224103ee0 msgr2=0x7f32241954c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224103ee0 0x7f32241954c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 -- 172.21.15.64:0/2657778817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32240ff7e0 msgr2=0x7f3224194f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32240ff7e0 0x7f3224194f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 -- 172.21.15.64:0/2657778817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32180095f0 con 0x7f32240fee40 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f322a150700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f32140060b0 tx=0x7f321400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.092+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400ddd0 con 0x7f32240fee40 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.093+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32241a4890 con 0x7f32240fee40 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.093+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f321400f040 con 0x7f32240fee40 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.093+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3214014600 con 0x7f32240fee40 2023-07-14T18:24:00.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.093+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32241a4d60 con 0x7f32240fee40 2023-07-14T18:24:00.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.094+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322404e340 con 0x7f32240fee40 2023-07-14T18:24:00.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.095+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3214015400 con 0x7f32240fee40 2023-07-14T18:24:00.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.095+0000 7f32237fe700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 0x7f320c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.095+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f32140cc630 con 0x7f32240fee40 2023-07-14T18:24:00.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.095+0000 7f322994f700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 0x7f320c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.096+0000 7f322994f700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 0x7f320c0a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3218000c00 tx=0x7f3218003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:00.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.100+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f321404f780 con 0x7f32240fee40 2023-07-14T18:24:00.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 -- 172.21.15.64:0/323716529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 msgr2=0x7fc95810c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 --2- 172.21.15.64:0/323716529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95810c6b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc95000b0a0 tx=0x7fc95000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 -- 172.21.15.64:0/323716529 shutdown_connections 2023-07-14T18:24:00.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 --2- 172.21.15.64:0/323716529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95810c6b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 --2- 172.21.15.64:0/323716529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958100f20 0x7fc9581012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 --2- 172.21.15.64:0/323716529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958101830 0x7fc958109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.257+0000 7fc95faf1700 1 -- 172.21.15.64:0/323716529 >> 172.21.15.64:0/323716529 conn(0x7fc958075ca0 msgr2=0x7fc9580760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:00.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- 172.21.15.64:0/323716529 shutdown_connections 2023-07-14T18:24:00.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- 172.21.15.64:0/323716529 wait complete. 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 Processor -- start 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- start start 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958100f20 0x7fc958195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 0x7fc9581962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819ad70 con 0x7fc95810a270 2023-07-14T18:24:00.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819aee0 con 0x7fc958101830 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.258+0000 7fc95faf1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95819b0a0 con 0x7fc958100f20 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47784/0 (socket says 172.21.15.64:47784) 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 -- 172.21.15.64:0/798936136 learned_addr learned my addr 172.21.15.64:0/798936136 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95d08c700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 0x7fc9581962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95d88d700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958100f20 0x7fc958195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 -- 172.21.15.64:0/798936136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958100f20 msgr2=0x7fc958195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958100f20 0x7fc958195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 -- 172.21.15.64:0/798936136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 msgr2=0x7fc9581962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 0x7fc9581962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 -- 172.21.15.64:0/798936136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc950009d00 con 0x7fc95810a270 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95d08c700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 0x7fc9581962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95e08e700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc95000b780 tx=0x7fc950008ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000e040 con 0x7fc95810a270 2023-07-14T18:24:00.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc95001d470 con 0x7fc95810a270 2023-07-14T18:24:00.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95819b320 con 0x7fc95810a270 2023-07-14T18:24:00.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.259+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9500052a0 con 0x7fc95810a270 2023-07-14T18:24:00.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.260+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9581a2be0 con 0x7fc95810a270 2023-07-14T18:24:00.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.261+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95804e340 con 0x7fc95810a270 2023-07-14T18:24:00.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.263+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc95001d5e0 con 0x7fc95810a270 2023-07-14T18:24:00.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.263+0000 7fc94effd700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 0x7fc9440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:00.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.263+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc9500cdf50 con 0x7fc95810a270 2023-07-14T18:24:00.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.263+0000 7fc95d88d700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 0x7fc9440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:00.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.264+0000 7fc95d88d700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 0x7fc9440a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc954009c60 tx=0x7fc954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:00.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.268+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc9500973e0 con 0x7fc95810a270 2023-07-14T18:24:00.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.311+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f32241a4fb0 con 0x7f32240fee40 2023-07-14T18:24:00.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.311+0000 7f32237fe700 1 -- 172.21.15.64:0/2657778817 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3214095a30 con 0x7f32240fee40 2023-07-14T18:24:00.365 INFO:teuthology.orchestra.run.smithi064.stdout:55834574913 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 msgr2=0x7f320c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 0x7f320c0a3690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f3218000c00 tx=0x7f3218003680 comp rx=0 tx=0).stop 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 msgr2=0x7f3224194a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f32140060b0 tx=0x7f321400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 shutdown_connections 2023-07-14T18:24:00.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f3224103ee0 0x7f32241954c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f320c0a11e0 0x7f320c0a3690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32240fee40 0x7f3224194a40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 --2- 172.21.15.64:0/2657778817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32240ff7e0 0x7f3224194f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 >> 172.21.15.64:0/2657778817 conn(0x7f3224075ca0 msgr2=0x7f3224102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:00.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 shutdown_connections 2023-07-14T18:24:00.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.313+0000 7f322c3b4700 1 -- 172.21.15.64:0/2657778817 wait complete. 2023-07-14T18:24:00.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.477+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fc9581a2d70 con 0x7fc95810a270 2023-07-14T18:24:00.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.477+0000 7fc94effd700 1 -- 172.21.15.64:0/798936136 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc95009aa00 con 0x7fc95810a270 2023-07-14T18:24:00.530 INFO:teuthology.orchestra.run.smithi064.stdout:274877906975 2023-07-14T18:24:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 msgr2=0x7fc9440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 0x7fc9440a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc954009c60 tx=0x7fc954009260 comp rx=0 tx=0).stop 2023-07-14T18:24:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 msgr2=0x7fc95819a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc95000b780 tx=0x7fc950008ec0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 shutdown_connections 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc958100f20 0x7fc958195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9440a11e0 0x7fc9440a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc95810a270 0x7fc95819a760 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 --2- 172.21.15.64:0/798936136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc958101830 0x7fc9581962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 >> 172.21.15.64:0/798936136 conn(0x7fc958075ca0 msgr2=0x7fc95810a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 shutdown_connections 2023-07-14T18:24:00.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:00.480+0000 7fc95faf1700 1 -- 172.21.15.64:0/798936136 wait complete. 2023-07-14T18:24:00.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:00 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1005360022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:24:00.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:00 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2657778817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:24:00.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:00 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1005360022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:24:00.782 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:00 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2657778817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:24:00.801 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514516 got 326417514517 for osd.11 2023-07-14T18:24:00.801 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:00 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1005360022' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T18:24:00.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:00 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2657778817' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T18:24:00.879 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920818 got 141733920820 for osd.4 2023-07-14T18:24:00.880 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:01.498 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574910 got 55834574913 for osd.1 2023-07-14T18:24:01.498 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:01.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:01 smithi144 ceph-mon[98417]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:01.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/798936136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:24:01.707 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:01 smithi064 ceph-mon[96720]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:01.708 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/798936136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:24:01.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:01 smithi105 ceph-mon[98013]: pgmap v282: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:01.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/798936136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T18:24:01.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/343115031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d18101150 msgr2=0x7f8d1810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:01.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/343115031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d18101150 0x7f8d1810bb40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b0d0 tx=0x7f8d1000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/343115031 shutdown_connections 2023-07-14T18:24:01.916 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/343115031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d18101150 0x7f8d1810bb40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/343115031 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d180fef40 0x7f8d180ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/343115031 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d181007a0 0x7f8d18100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/343115031 >> 172.21.15.64:0/343115031 conn(0x7f8d18075c90 msgr2=0x7f8d18076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/343115031 shutdown_connections 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.862+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/343115031 wait complete. 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 Processor -- start 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 -- start start 2023-07-14T18:24:01.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d181007a0 0x7f8d18197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d18101150 0x7f8d181a1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d181a22b0 con 0x7f8d180fef40 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d181a23f0 con 0x7f8d181007a0 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.863+0000 7f8d1ffaf700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d181a2530 con 0x7f8d18101150 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:01.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47794/0 (socket says 172.21.15.64:47794) 2023-07-14T18:24:01.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 -- 172.21.15.64:0/1109442918 learned_addr learned my addr 172.21.15.64:0/1109442918 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:01.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1d54a700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d181007a0 0x7f8d18197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:01.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1e54c700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d18101150 0x7f8d181a1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:01.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 -- 172.21.15.64:0/1109442918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d18101150 msgr2=0x7f8d181a1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:01.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d18101150 0x7f8d181a1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 -- 172.21.15.64:0/1109442918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d181007a0 msgr2=0x7f8d18197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:01.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d181007a0 0x7f8d18197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 -- 172.21.15.64:0/1109442918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d10009d00 con 0x7f8d180fef40 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1dd4b700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d9b0 tx=0x7f8d1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14004220 con 0x7f8d180fef40 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d18194c00 con 0x7f8d180fef40 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d18195150 con 0x7f8d180fef40 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.864+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8d14007640 con 0x7f8d180fef40 2023-07-14T18:24:01.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.865+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1400bc60 con 0x7f8d180fef40 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.865+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cfc005320 con 0x7f8d180fef40 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.867+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8d14004750 con 0x7f8d180fef40 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.867+0000 7f8d0effd700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 0x7f8d040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.867+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8d140cce10 con 0x7f8d180fef40 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.867+0000 7f8d1d54a700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 0x7f8d040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:01.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.868+0000 7f8d1d54a700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 0x7f8d040a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8d180695e0 tx=0x7f8d080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:01.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.874+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8d14096210 con 0x7f8d180fef40 2023-07-14T18:24:01.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 -- 172.21.15.64:0/3377812248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc0fff40 msgr2=0x7fe4cc100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 --2- 172.21.15.64:0/3377812248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc100310 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0099c0 tx=0x7fe4bc009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 -- 172.21.15.64:0/3377812248 shutdown_connections 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 --2- 172.21.15.64:0/3377812248 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc10cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 --2- 172.21.15.64:0/3377812248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc100310 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 --2- 172.21.15.64:0/3377812248 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc1008e0 0x7fe4cc1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:01.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 -- 172.21.15.64:0/3377812248 >> 172.21.15.64:0/3377812248 conn(0x7fe4cc0fb7c0 msgr2=0x7fe4cc0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:01.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 -- 172.21.15.64:0/3377812248 shutdown_connections 2023-07-14T18:24:01.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.944+0000 7fe4d149d700 1 -- 172.21.15.64:0/3377812248 wait complete. 2023-07-14T18:24:01.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.945+0000 7fe4d149d700 1 Processor -- start 2023-07-14T18:24:01.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.945+0000 7fe4d149d700 1 -- start start 2023-07-14T18:24:01.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc1008e0 0x7fe4cc1961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:01.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc199ba0 con 0x7fe4cc104fe0 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc199d10 con 0x7fe4cc0fff40 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4d149d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4cc199ed0 con 0x7fe4cc1008e0 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.64:41346/0 (socket says 172.21.15.64:41346) 2023-07-14T18:24:02.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4cb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47810/0 (socket says 172.21.15.64:47810) 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 -- 172.21.15.64:0/4025853945 learned_addr learned my addr 172.21.15.64:0/4025853945 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4ca7fc700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc1008e0 0x7fe4cc1961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 -- 172.21.15.64:0/4025853945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc1008e0 msgr2=0x7fe4cc1961e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc1008e0 0x7fe4cc1961e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 -- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 msgr2=0x7fe4cc1995a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.946+0000 7fe4caffd700 1 -- 172.21.15.64:0/4025853945 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc0095f0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4cb7fe700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:02.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4caffd700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005910 tx=0x7fe4bc009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc01d070 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4d149d700 1 -- 172.21.15.64:0/4025853945 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4cc19a150 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe4bc022470 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4d149d700 1 -- 172.21.15.64:0/4025853945 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4cc072a40 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.947+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc017650 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.948+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac0052f0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.949+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe4bc0225e0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.950+0000 7fe4b3fff700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 0x7fe4b40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.951+0000 7fe4ca7fc700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 0x7fe4b40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.951+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe4bc0cdda0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.951+0000 7fe4ca7fc700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 0x7fe4b40a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc197280 tx=0x7fe4c0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:01.954+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe4bc0971d0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.021+0000 7efdacbb4700 1 -- 172.21.15.64:0/2570934645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 msgr2=0x7efda81012c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.021+0000 7efdacbb4700 1 --2- 172.21.15.64:0/2570934645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda81012c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7efd98009990 tx=0x7efd98009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 -- 172.21.15.64:0/2570934645 shutdown_connections 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 --2- 172.21.15.64:0/2570934645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda810c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 --2- 172.21.15.64:0/2570934645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda81012c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 --2- 172.21.15.64:0/2570934645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 0x7efda8109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 -- 172.21.15.64:0/2570934645 >> 172.21.15.64:0/2570934645 conn(0x7efda8075d20 msgr2=0x7efda8076120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 -- 172.21.15.64:0/2570934645 shutdown_connections 2023-07-14T18:24:02.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.022+0000 7efdacbb4700 1 -- 172.21.15.64:0/2570934645 wait complete. 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 Processor -- start 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 -- start start 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 0x7efda80ff090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda80ff610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda81a4150 con 0x7efda8100ef0 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda81a42c0 con 0x7efda810a2d0 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efdacbb4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda81a4480 con 0x7efda8101800 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efda659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efda659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47822/0 (socket says 172.21.15.64:47822) 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.023+0000 7efda659c700 1 -- 172.21.15.64:0/572063296 learned_addr learned my addr 172.21.15.64:0/572063296 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda6d9d700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda80ff610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda5d9b700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 0x7efda80ff090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 -- 172.21.15.64:0/572063296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 msgr2=0x7efda80ff090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 0x7efda80ff090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 -- 172.21.15.64:0/572063296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 msgr2=0x7efda80ff610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda80ff610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 -- 172.21.15.64:0/572063296 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd980095f0 con 0x7efda8100ef0 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda6d9d700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda80ff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efda659c700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efd98005910 tx=0x7efd98009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.079 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9801d070 con 0x7efda8100ef0 2023-07-14T18:24:02.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda81a4700 con 0x7efda8100ef0 2023-07-14T18:24:02.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efd98003f40 con 0x7efda8100ef0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9800fc60 con 0x7efda8100ef0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.024+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda81a4ba0 con 0x7efda8100ef0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.026+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda8104b30 con 0x7efda8100ef0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.028+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efd980040b0 con 0x7efda8100ef0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.028+0000 7efd977fe700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 0x7efd900a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.029+0000 7efda5d9b700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 0x7efd900a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.081 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.029+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efd980cdea0 con 0x7efda8100ef0 2023-07-14T18:24:02.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.029+0000 7efda5d9b700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 0x7efd900a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efd9c005950 tx=0x7efd9c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.035+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efd98097330 con 0x7efda8100ef0 2023-07-14T18:24:02.097 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 -- 172.21.15.64:0/3993887498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad140febc0 msgr2=0x7fad1410cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.097 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 --2- 172.21.15.64:0/3993887498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad140febc0 0x7fad1410cb30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fad0c00b1a0 tx=0x7fad0c00b4b0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 -- 172.21.15.64:0/3993887498 shutdown_connections 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 --2- 172.21.15.64:0/3993887498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad140febc0 0x7fad1410cb30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 --2- 172.21.15.64:0/3993887498 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad14104c70 0x7fad14105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 --2- 172.21.15.64:0/3993887498 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad140fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 -- 172.21.15.64:0/3993887498 >> 172.21.15.64:0/3993887498 conn(0x7fad14075280 msgr2=0x7fad14075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 -- 172.21.15.64:0/3993887498 shutdown_connections 2023-07-14T18:24:02.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.045+0000 7fad19959700 1 -- 172.21.15.64:0/3993887498 wait complete. 2023-07-14T18:24:02.099 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.046+0000 7fad19959700 1 Processor -- start 2023-07-14T18:24:02.099 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.046+0000 7fad19959700 1 -- start start 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad140febc0 0x7fad14199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad14104c70 0x7fad14193830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14193fe0 con 0x7fad14104c70 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14194150 con 0x7fad140febc0 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad19959700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14194310 con 0x7fad140fe210 2023-07-14T18:24:02.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:55492/0 (socket says 172.21.15.64:55492) 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 -- 172.21.15.64:0/1438258400 learned_addr learned my addr 172.21.15.64:0/1438258400 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad127fc700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad140febc0 0x7fad14199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad137fe700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad14104c70 0x7fad14193830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 -- 172.21.15.64:0/1438258400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad140febc0 msgr2=0x7fad14199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad140febc0 0x7fad14199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 -- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad14104c70 msgr2=0x7fad14193830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad14104c70 0x7fad14193830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 -- 172.21.15.64:0/1438258400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0c009d00 con 0x7fad140fe210 2023-07-14T18:24:02.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.047+0000 7fad12ffd700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad0400d950 tx=0x7fad0400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.048+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0400f890 con 0x7fad140fe210 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.048+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad141945f0 con 0x7fad140fe210 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.048+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fad0400fe80 con 0x7fad140fe210 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.048+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad141a0810 con 0x7fad140fe210 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.048+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad0400e3b0 con 0x7fad140fe210 2023-07-14T18:24:02.103 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.049+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000052f0 con 0x7fad140fe210 2023-07-14T18:24:02.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.050+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fad0400f9f0 con 0x7fad140fe210 2023-07-14T18:24:02.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.051+0000 7fad18957700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 0x7facfc0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:02.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.051+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fad040ccbf0 con 0x7fad140fe210 2023-07-14T18:24:02.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.051+0000 7fad127fc700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 0x7facfc0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:02.104 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.051+0000 7fad127fc700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 0x7facfc0a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad14195060 tx=0x7fad080073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:02.106 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.054+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fad04095ff0 con 0x7fad140fe210 2023-07-14T18:24:02.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.112+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8cfc005cc0 con 0x7f8d180fef40 2023-07-14T18:24:02.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.113+0000 7f8d0effd700 1 -- 172.21.15.64:0/1109442918 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8d14016020 con 0x7f8d180fef40 2023-07-14T18:24:02.219 INFO:teuthology.orchestra.run.smithi064.stdout:214748364839 2023-07-14T18:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 msgr2=0x7f8d040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 0x7f8d040a3710 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8d180695e0 tx=0x7f8d080073d0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 msgr2=0x7f8d18196cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d9b0 tx=0x7f8d1400dcc0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 shutdown_connections 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8d18101150 0x7f8d181a1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8d040a1260 0x7f8d040a3710 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d180fef40 0x7f8d18196cf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 --2- 172.21.15.64:0/1109442918 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d181007a0 0x7f8d18197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 >> 172.21.15.64:0/1109442918 conn(0x7f8d18075c90 msgr2=0x7f8d180fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 shutdown_connections 2023-07-14T18:24:02.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.116+0000 7f8d1ffaf700 1 -- 172.21.15.64:0/1109442918 wait complete. 2023-07-14T18:24:02.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.175+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe4ac005c90 con 0x7fe4cc0fff40 2023-07-14T18:24:02.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.176+0000 7fe4b3fff700 1 -- 172.21.15.64:0/4025853945 <== mon.2 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe4bc09a7f0 con 0x7fe4cc0fff40 2023-07-14T18:24:02.228 INFO:teuthology.orchestra.run.smithi064.stdout:107374182456 2023-07-14T18:24:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 msgr2=0x7fe4b40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 0x7fe4b40a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc197280 tx=0x7fe4c0009260 comp rx=0 tx=0).stop 2023-07-14T18:24:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 msgr2=0x7fe4cc195ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005910 tx=0x7fe4bc009580 comp rx=0 tx=0).stop 2023-07-14T18:24:02.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 shutdown_connections 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe4b40a11e0 0x7fe4b40a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe4cc104fe0 0x7fe4cc1995a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe4cc0fff40 0x7fe4cc195ca0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 --2- 172.21.15.64:0/4025853945 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe4cc1008e0 0x7fe4cc1961e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 >> 172.21.15.64:0/4025853945 conn(0x7fe4cc0fb7c0 msgr2=0x7fe4cc0fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 shutdown_connections 2023-07-14T18:24:02.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.179+0000 7fe4b1ffb700 1 -- 172.21.15.64:0/4025853945 wait complete. 2023-07-14T18:24:02.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.296+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7efda804e340 con 0x7efda8100ef0 2023-07-14T18:24:02.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.296+0000 7efd977fe700 1 -- 172.21.15.64:0/572063296 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7efd9809a950 con 0x7efda8100ef0 2023-07-14T18:24:02.349 INFO:teuthology.orchestra.run.smithi064.stdout:244813135907 2023-07-14T18:24:02.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 msgr2=0x7efd900a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 0x7efd900a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7efd9c005950 tx=0x7efd9c0058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 msgr2=0x7efda80feb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7efd98005910 tx=0x7efd98009580 comp rx=0 tx=0).stop 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 shutdown_connections 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7efd900a11e0 0x7efd900a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efda810a2d0 0x7efda80ff610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efda8100ef0 0x7efda80feb50 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 --2- 172.21.15.64:0/572063296 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7efda8101800 0x7efda80ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 >> 172.21.15.64:0/572063296 conn(0x7efda8075d20 msgr2=0x7efda8101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 shutdown_connections 2023-07-14T18:24:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7efdacbb4700 1 -- 172.21.15.64:0/572063296 wait complete. 2023-07-14T18:24:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.298+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fad00005c90 con 0x7fad140fe210 2023-07-14T18:24:02.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.299+0000 7fad18957700 1 -- 172.21.15.64:0/1438258400 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fad04020020 con 0x7fad140fe210 2023-07-14T18:24:02.353 INFO:teuthology.orchestra.run.smithi064.stdout:34359738437 2023-07-14T18:24:02.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 msgr2=0x7facfc0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 0x7facfc0a3710 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fad14195060 tx=0x7fad080073d0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 msgr2=0x7fad14199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fad0400d950 tx=0x7fad0400dc60 comp rx=0 tx=0).stop 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 shutdown_connections 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fad140fe210 0x7fad14199840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7facfc0a1260 0x7facfc0a3710 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fad14104c70 0x7fad14193830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 --2- 172.21.15.64:0/1438258400 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fad140febc0 0x7fad14199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:02.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 >> 172.21.15.64:0/1438258400 conn(0x7fad14075280 msgr2=0x7fad140fca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:02.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 shutdown_connections 2023-07-14T18:24:02.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:02.303+0000 7fad19959700 1 -- 172.21.15.64:0/1438258400 wait complete. 2023-07-14T18:24:02.539 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1109442918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:24:02.539 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4025853945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:24:02.540 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/572063296' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:24:02.540 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1438258400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:24:02.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1109442918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:24:02.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4025853945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:24:02.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/572063296' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:24:02.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1438258400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:24:02.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1109442918' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T18:24:02.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4025853945' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T18:24:02.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/572063296' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T18:24:02.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1438258400' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T18:24:02.954 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906972 got 274877906975 for osd.9 2023-07-14T18:24:02.955 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:03.355 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135905 got 244813135907 for osd.8 2023-07-14T18:24:03.356 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:03.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:03 smithi064 ceph-mon[96720]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T18:24:03.658 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182454 got 107374182456 for osd.3 2023-07-14T18:24:03.659 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:03.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:03 smithi105 ceph-mon[98013]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T18:24:03.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738436 got 34359738437 for osd.0 2023-07-14T18:24:03.837 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:03.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 -- 172.21.15.64:0/1836115611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400feb30 msgr2=0x7f294010cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 --2- 172.21.15.64:0/1836115611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400feb30 0x7f294010cb70 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f293800b0a0 tx=0x7f293800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 -- 172.21.15.64:0/1836115611 shutdown_connections 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 --2- 172.21.15.64:0/1836115611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400feb30 0x7f294010cb70 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 --2- 172.21.15.64:0/1836115611 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29401050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 --2- 172.21.15.64:0/1836115611 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400fe180 0x7f29400fe5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 -- 172.21.15.64:0/1836115611 >> 172.21.15.64:0/1836115611 conn(0x7f2940075050 msgr2=0x7f2940075450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:03.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 -- 172.21.15.64:0/1836115611 shutdown_connections 2023-07-14T18:24:03.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.829+0000 7f29452a1700 1 -- 172.21.15.64:0/1836115611 wait complete. 2023-07-14T18:24:03.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 Processor -- start 2023-07-14T18:24:03.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 -- start start 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400feb30 0x7f29400ff8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29400ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401a42b0 con 0x7f29400fe180 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401a4420 con 0x7f2940104cd0 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f29452a1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29401a45e0 con 0x7f29400feb30 2023-07-14T18:24:03.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f293effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f293effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47850/0 (socket says 172.21.15.64:47850) 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.830+0000 7f293effd700 1 -- 172.21.15.64:0/3371912971 learned_addr learned my addr 172.21.15.64:0/3371912971 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293e7fc700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400feb30 0x7f29400ff8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293f7fe700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29400ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 -- 172.21.15.64:0/3371912971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400feb30 msgr2=0x7f29400ff8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400feb30 0x7f29400ff8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 -- 172.21.15.64:0/3371912971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 msgr2=0x7f29400ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:03.886 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29400ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 -- 172.21.15.64:0/3371912971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2938009d00 con 0x7f29400fe180 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293f7fe700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29400ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f293effd700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f292800d950 tx=0x7f292800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2928009b40 con 0x7f29400fe180 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2928009ca0 con 0x7f29400fe180 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29401a48c0 con 0x7f29400fe180 2023-07-14T18:24:03.887 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292800f620 con 0x7f29400fe180 2023-07-14T18:24:03.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.831+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29401a4dc0 con 0x7f29400fe180 2023-07-14T18:24:03.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.832+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294004e340 con 0x7f29400fe180 2023-07-14T18:24:03.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.833+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2928010460 con 0x7f29400fe180 2023-07-14T18:24:03.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.834+0000 7f2937fff700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 0x7f292c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:03.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.834+0000 7f293e7fc700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 0x7f292c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:03.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.834+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f29280cd040 con 0x7f29400fe180 2023-07-14T18:24:03.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.834+0000 7f293e7fc700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 0x7f292c0a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f29300058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:03.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:03.838+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2928096440 con 0x7f29400fe180 2023-07-14T18:24:03.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:03 smithi144 ceph-mon[98417]: pgmap v283: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 170 B/s wr, 8 op/s 2023-07-14T18:24:03.971 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364836 got 214748364839 for osd.7 2023-07-14T18:24:03.971 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:04.097 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.045+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f29401a4fe0 con 0x7f29400fe180 2023-07-14T18:24:04.098 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.045+0000 7f2937fff700 1 -- 172.21.15.64:0/3371912971 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f2928099a60 con 0x7f29400fe180 2023-07-14T18:24:04.098 INFO:teuthology.orchestra.run.smithi064.stdout:296352743450 2023-07-14T18:24:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 msgr2=0x7f292c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 0x7f292c0a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f29300058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 msgr2=0x7f29400ff360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.100 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f292800d950 tx=0x7f292800dd10 comp rx=0 tx=0).stop 2023-07-14T18:24:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 shutdown_connections 2023-07-14T18:24:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f292c0a11e0 0x7f292c0a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2940104cd0 0x7f29400ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29400fe180 0x7f29400ff360 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.101 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 --2- 172.21.15.64:0/3371912971 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29400feb30 0x7f29400ff8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.047+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 >> 172.21.15.64:0/3371912971 conn(0x7f2940075050 msgr2=0x7f29400fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.048+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 shutdown_connections 2023-07-14T18:24:04.102 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.048+0000 7f29452a1700 1 -- 172.21.15.64:0/3371912971 wait complete. 2023-07-14T18:24:04.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 -- 172.21.15.64:0/2212053480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78069770 msgr2=0x7fbb781048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/2212053480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78069770 0x7fbb781048c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbb6c0099c0 tx=0x7fbb6c009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 -- 172.21.15.64:0/2212053480 shutdown_connections 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/2212053480 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78068dd0 0x7fbb780691a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/2212053480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78104ff0 0x7fbb7810cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/2212053480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78069770 0x7fbb781048c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 -- 172.21.15.64:0/2212053480 >> 172.21.15.64:0/2212053480 conn(0x7fbb78075a80 msgr2=0x7fbb78075e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 -- 172.21.15.64:0/2212053480 shutdown_connections 2023-07-14T18:24:04.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.111+0000 7fbb7df0e700 1 -- 172.21.15.64:0/2212053480 wait complete. 2023-07-14T18:24:04.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 Processor -- start 2023-07-14T18:24:04.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 -- start start 2023-07-14T18:24:04.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 0x7fbb78195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:04.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:04.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 0x7fbb7819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7819acf0 con 0x7fbb78068dd0 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7819ae60 con 0x7fbb78069770 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.112+0000 7fbb7df0e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7819b020 con 0x7fbb78104ff0 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb76ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb76ffd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.64:41404/0 (socket says 172.21.15.64:41404) 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb76ffd700 1 -- 172.21.15.64:0/383985619 learned_addr learned my addr 172.21.15.64:0/383985619 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:04.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 0x7fbb7819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb777fe700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 0x7fbb78195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 -- 172.21.15.64:0/383985619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 msgr2=0x7fbb78196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 -- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 msgr2=0x7fbb78195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 0x7fbb78195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 -- 172.21.15.64:0/383985619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c0095f0 con 0x7fbb78104ff0 2023-07-14T18:24:04.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb76ffd700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb777fe700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 0x7fbb78195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.113+0000 7fbb77fff700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 0x7fbb7819a6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000cc50 tx=0x7fbb7000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.114+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb7000aa90 con 0x7fbb78104ff0 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.114+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbb7000abf0 con 0x7fbb78104ff0 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.114+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb700115e0 con 0x7fbb78104ff0 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.114+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb7819b300 con 0x7fbb78104ff0 2023-07-14T18:24:04.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.114+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb781a2b60 con 0x7fbb78104ff0 2023-07-14T18:24:04.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.115+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb7804e340 con 0x7fbb78104ff0 2023-07-14T18:24:04.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.117+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbb70004ad0 con 0x7fbb78104ff0 2023-07-14T18:24:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.117+0000 7fbb74ff9700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 0x7fbb600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.117+0000 7fbb777fe700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 0x7fbb600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.117+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbb700cccc0 con 0x7fbb78104ff0 2023-07-14T18:24:04.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.118+0000 7fbb777fe700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 0x7fbb600a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb68011490 tx=0x7fbb68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:04.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.121+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbb700961d0 con 0x7fbb78104ff0 2023-07-14T18:24:04.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.330+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbb78197130 con 0x7fbb78104ff0 2023-07-14T18:24:04.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.330+0000 7fbb74ff9700 1 -- 172.21.15.64:0/383985619 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbb70016070 con 0x7fbb78104ff0 2023-07-14T18:24:04.383 INFO:teuthology.orchestra.run.smithi064.stdout:77309411387 2023-07-14T18:24:04.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 msgr2=0x7fbb600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 0x7fbb600a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb68011490 tx=0x7fbb68009260 comp rx=0 tx=0).stop 2023-07-14T18:24:04.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 msgr2=0x7fbb7819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 0x7fbb7819a6e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000cc50 tx=0x7fbb7000cf60 comp rx=0 tx=0).stop 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 shutdown_connections 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbb78104ff0 0x7fbb7819a6e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbb600a1260 0x7fbb600a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb78068dd0 0x7fbb78195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 --2- 172.21.15.64:0/383985619 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb78069770 0x7fbb78196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:04.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 >> 172.21.15.64:0/383985619 conn(0x7fbb78075a80 msgr2=0x7fbb780fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:04.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 shutdown_connections 2023-07-14T18:24:04.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:04.333+0000 7fbb7df0e700 1 -- 172.21.15.64:0/383985619 wait complete. 2023-07-14T18:24:04.536 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743449 got 296352743450 for osd.10 2023-07-14T18:24:04.537 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:04.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3371912971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:24:04.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/383985619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:24:04.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3371912971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:24:04.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/383985619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:24:04.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411385 got 77309411387 for osd.2 2023-07-14T18:24:04.905 DEBUG:teuthology.parallel:result is None 2023-07-14T18:24:04.905 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T18:24:04.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:24:04.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3371912971' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T18:24:04.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/383985619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T18:24:05.471 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:05.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:05 smithi105 ceph-mon[98013]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2023-07-14T18:24:05.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:05 smithi064 ceph-mon[96720]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2023-07-14T18:24:05.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:05 smithi144 ceph-mon[98417]: pgmap v284: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 170 B/s wr, 11 op/s 2023-07-14T18:24:06.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:24:06.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:24:06.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2023-07-14T18:24:06.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:06 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:24:06.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:24:06.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T18:24:06.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T18:24:06.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-14T18:24:06.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2023-07-14T18:24:06.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2023-07-14T18:24:06.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:24:06.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2023-07-14T18:24:06.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:06 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:24:06.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]: dispatch 2023-07-14T18:24:06.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]: dispatch 2023-07-14T18:24:06.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T18:24:06.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]: dispatch 2023-07-14T18:24:06.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2023-07-14T18:24:06.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]: dispatch 2023-07-14T18:24:06.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]: dispatch 2023-07-14T18:24:06.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2023-07-14T18:24:06.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]: dispatch 2023-07-14T18:24:06.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:06 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:24:06.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 -- 172.21.15.64:0/896680284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5788105da0 msgr2=0x7f5788106170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 --2- 172.21.15.64:0/896680284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5788105da0 0x7f5788106170 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f57780099c0 tx=0x7f5778009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 -- 172.21.15.64:0/896680284 shutdown_connections 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 --2- 172.21.15.64:0/896680284 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 0x7f578810cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 --2- 172.21.15.64:0/896680284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5788105da0 0x7f5788106170 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 --2- 172.21.15.64:0/896680284 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f57880fe3a0 0x7f57880fe810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 -- 172.21.15.64:0/896680284 >> 172.21.15.64:0/896680284 conn(0x7f5788076370 msgr2=0x7f5788076770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:06.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 -- 172.21.15.64:0/896680284 shutdown_connections 2023-07-14T18:24:06.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.938+0000 7f578ce6a700 1 -- 172.21.15.64:0/896680284 wait complete. 2023-07-14T18:24:06.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 Processor -- start 2023-07-14T18:24:06.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 -- start start 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 0x7f578819bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 0x7f5788195a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57881960b0 con 0x7f57880fe3a0 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5788196220 con 0x7f57880fed50 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578ce6a700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57881963e0 con 0x7f5788105da0 2023-07-14T18:24:06.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578659c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47876/0 (socket says 172.21.15.64:47876) 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f578659c700 1 -- 172.21.15.64:0/2929429615 learned_addr learned my addr 172.21.15.64:0/2929429615 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f5786d9d700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 0x7f5788195a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.939+0000 7f5785d9b700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 0x7f578819bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 -- 172.21.15.64:0/2929429615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 msgr2=0x7f578819bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 0x7f578819bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 -- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 msgr2=0x7f578819b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 -- 172.21.15.64:0/2929429615 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57780095f0 con 0x7f5788105da0 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f578659c700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f5786d9d700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 0x7f5788195a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f578000cc50 tx=0x7f578000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578000aa90 con 0x7f5788105da0 2023-07-14T18:24:06.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f578000abf0 con 0x7f5788105da0 2023-07-14T18:24:06.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57800115e0 con 0x7f5788105da0 2023-07-14T18:24:06.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57881966c0 con 0x7f5788105da0 2023-07-14T18:24:06.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.940+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57881a29b0 con 0x7f5788105da0 2023-07-14T18:24:06.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.941+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578804e340 con 0x7f5788105da0 2023-07-14T18:24:06.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.943+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5780012420 con 0x7f5788105da0 2023-07-14T18:24:06.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.944+0000 7f57777fe700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 0x7f57700a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:06.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.944+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f57800cd270 con 0x7f5788105da0 2023-07-14T18:24:06.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.944+0000 7f578659c700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 0x7f57700a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:06.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.944+0000 7f578659c700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 0x7f57700a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5778005b20 tx=0x7f577801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:06.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:06.946+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5780096660 con 0x7f5788105da0 2023-07-14T18:24:07.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.145+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5788104120 con 0x7f57700a1260 2023-07-14T18:24:07.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.159+0000 7f57777fe700 1 -- 172.21.15.64:0/2929429615 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423156 (secure 0 0 0) 0x7f5788104120 con 0x7f57700a1260 2023-07-14T18:24:07.219 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:07.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.163+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 msgr2=0x7f57700a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:07.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.163+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 0x7f57700a3710 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5778005b20 tx=0x7f577801a040 comp rx=0 tx=0).stop 2023-07-14T18:24:07.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.163+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 msgr2=0x7f5788195a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:07.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.163+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 0x7f5788195a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f578000cc50 tx=0x7f578000cf60 comp rx=0 tx=0).stop 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 shutdown_connections 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5788105da0 0x7f5788195a40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f57700a1260 0x7f57700a3710 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f57880fe3a0 0x7f578819b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 --2- 172.21.15.64:0/2929429615 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57880fed50 0x7f578819bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 >> 172.21.15.64:0/2929429615 conn(0x7f5788076370 msgr2=0x7f5788103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:07.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 shutdown_connections 2023-07-14T18:24:07.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:07.164+0000 7f578ce6a700 1 -- 172.21.15.64:0/2929429615 wait complete. 2023-07-14T18:24:07.224 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:24:07.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:24:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T18:24:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T18:24:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-14T18:24:07.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2023-07-14T18:24:07.594 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:07 smithi064 ceph-mon[96720]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:24:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:24:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T18:24:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-07-14T18:24:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2023-07-14T18:24:07.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:07 smithi105 ceph-mon[98013]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:24:07.850 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2023-07-14T18:24:06.400616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":523,"num_read_kb":382,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337388,"kb_used_data":13196,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604436,"statfs":{"total":1151940427776,"available":1151594942464,"internally_reserved":0,"allocated":13512704,"data_stored":7433939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1652,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":97,"num_read_kb":49,"num_write":1,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.930547"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227165+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:23:11.227165+0000","last_peered":"2023-07-14T18:23:11.227165+0000","last_clean":"2023-07-14T18:23:11.227165+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:23:11.227165+0000","last_undegraded":"2023-07-14T18:23:11.227165+0000","last_fullsized":"2023-07-14T18:23:11.227165+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:09.995395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226781+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:23:11.226781+0000","last_peered":"2023-07-14T18:23:11.226781+0000","last_clean":"2023-07-14T18:23:11.226781+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:23:11.226781+0000","last_undegraded":"2023-07-14T18:23:11.226781+0000","last_fullsized":"2023-07-14T18:23:11.226781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:58.662673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226533+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:23:11.226533+0000","last_peered":"2023-07-14T18:23:11.226533+0000","last_clean":"2023-07-14T18:23:11.226533+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:23:11.226533+0000","last_undegraded":"2023-07-14T18:23:11.226533+0000","last_fullsized":"2023-07-14T18:23:11.226533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:50.079766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227979+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:23:11.227979+0000","last_peered":"2023-07-14T18:23:11.227979+0000","last_clean":"2023-07-14T18:23:11.227979+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:23:11.227979+0000","last_undegraded":"2023-07-14T18:23:11.227979+0000","last_fullsized":"2023-07-14T18:23:11.227979+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:06.130874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227740+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:23:11.227740+0000","last_peered":"2023-07-14T18:23:11.227740+0000","last_clean":"2023-07-14T18:23:11.227740+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:23:11.227740+0000","last_undegraded":"2023-07-14T18:23:11.227740+0000","last_fullsized":"2023-07-14T18:23:11.227740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:12.302519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227464+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:23:11.227464+0000","last_peered":"2023-07-14T18:23:11.227464+0000","last_clean":"2023-07-14T18:23:11.227464+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:23:11.227464+0000","last_undegraded":"2023-07-14T18:23:11.227464+0000","last_fullsized":"2023-07-14T18:23:11.227464+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:01.500398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610199+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:23:10.610199+0000","last_peered":"2023-07-14T18:23:10.610199+0000","last_clean":"2023-07-14T18:23:10.610199+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:23:10.610199+0000","last_undegraded":"2023-07-14T18:23:10.610199+0000","last_fullsized":"2023-07-14T18:23:10.610199+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:21:09.538683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226752+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:23:11.226752+0000","last_peered":"2023-07-14T18:23:11.226752+0000","last_clean":"2023-07-14T18:23:11.226752+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:23:11.226752+0000","last_undegraded":"2023-07-14T18:23:11.226752+0000","last_fullsized":"2023-07-14T18:23:11.226752+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:25.294010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.448033+0000","last_change":"2023-07-14T18:23:05.248889+0000","last_active":"2023-07-14T18:23:10.448033+0000","last_peered":"2023-07-14T18:23:10.448033+0000","last_clean":"2023-07-14T18:23:10.448033+0000","last_became_active":"2023-07-14T18:23:05.248500+0000","last_became_peered":"2023-07-14T18:23:05.248500+0000","last_unstale":"2023-07-14T18:23:10.448033+0000","last_undegraded":"2023-07-14T18:23:10.448033+0000","last_fullsized":"2023-07-14T18:23:10.448033+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:53:30.313031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228071+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:23:11.228071+0000","last_peered":"2023-07-14T18:23:11.228071+0000","last_clean":"2023-07-14T18:23:11.228071+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:23:11.228071+0000","last_undegraded":"2023-07-14T18:23:11.228071+0000","last_fullsized":"2023-07-14T18:23:11.228071+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:50:34.155670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610671+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:23:10.610671+0000","last_peered":"2023-07-14T18:23:10.610671+0000","last_clean":"2023-07-14T18:23:10.610671+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:23:10.610671+0000","last_undegraded":"2023-07-14T18:23:10.610671+0000","last_fullsized":"2023-07-14T18:23:10.610671+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:08.072924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227886+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:23:11.227886+0000","last_peered":"2023-07-14T18:23:11.227886+0000","last_clean":"2023-07-14T18:23:11.227886+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:23:11.227886+0000","last_undegraded":"2023-07-14T18:23:11.227886+0000","last_fullsized":"2023-07-14T18:23:11.227886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:15.483937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444815+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:23:10.444815+0000","last_peered":"2023-07-14T18:23:10.444815+0000","last_clean":"2023-07-14T18:23:10.444815+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:23:10.444815+0000","last_undegraded":"2023-07-14T18:23:10.444815+0000","last_fullsized":"2023-07-14T18:23:10.444815+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:06.195971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226320+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:23:11.226320+0000","last_peered":"2023-07-14T18:23:11.226320+0000","last_clean":"2023-07-14T18:23:11.226320+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:23:11.226320+0000","last_undegraded":"2023-07-14T18:23:11.226320+0000","last_fullsized":"2023-07-14T18:23:11.226320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:40.019875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226467+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:23:11.226467+0000","last_peered":"2023-07-14T18:23:11.226467+0000","last_clean":"2023-07-14T18:23:11.226467+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:23:11.226467+0000","last_undegraded":"2023-07-14T18:23:11.226467+0000","last_fullsized":"2023-07-14T18:23:11.226467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:46.851772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447060+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:23:10.447060+0000","last_peered":"2023-07-14T18:23:10.447060+0000","last_clean":"2023-07-14T18:23:10.447060+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:23:10.447060+0000","last_undegraded":"2023-07-14T18:23:10.447060+0000","last_fullsized":"2023-07-14T18:23:10.447060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:57.018695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227034+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:23:11.227034+0000","last_peered":"2023-07-14T18:23:11.227034+0000","last_clean":"2023-07-14T18:23:11.227034+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:23:11.227034+0000","last_undegraded":"2023-07-14T18:23:11.227034+0000","last_fullsized":"2023-07-14T18:23:11.227034+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:04.302893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444988+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:23:10.444988+0000","last_peered":"2023-07-14T18:23:10.444988+0000","last_clean":"2023-07-14T18:23:10.444988+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:23:10.444988+0000","last_undegraded":"2023-07-14T18:23:10.444988+0000","last_fullsized":"2023-07-14T18:23:10.444988+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:07.815568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445549+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:23:10.445549+0000","last_peered":"2023-07-14T18:23:10.445549+0000","last_clean":"2023-07-14T18:23:10.445549+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:23:10.445549+0000","last_undegraded":"2023-07-14T18:23:10.445549+0000","last_fullsized":"2023-07-14T18:23:10.445549+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:26.709513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445503+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:23:10.445503+0000","last_peered":"2023-07-14T18:23:10.445503+0000","last_clean":"2023-07-14T18:23:10.445503+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:23:10.445503+0000","last_undegraded":"2023-07-14T18:23:10.445503+0000","last_fullsized":"2023-07-14T18:23:10.445503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:03.591411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":69,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:01.346642+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:24:01.346642+0000","last_peered":"2023-07-14T18:24:01.346642+0000","last_clean":"2023-07-14T18:24:01.346642+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:24:01.346642+0000","last_undegraded":"2023-07-14T18:24:01.346642+0000","last_fullsized":"2023-07-14T18:24:01.346642+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:07:56.695338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":36,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227501+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:23:11.227501+0000","last_peered":"2023-07-14T18:23:11.227501+0000","last_clean":"2023-07-14T18:23:11.227501+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:23:11.227501+0000","last_undegraded":"2023-07-14T18:23:11.227501+0000","last_fullsized":"2023-07-14T18:23:11.227501+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:43:10.900906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226877+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:23:11.226877+0000","last_peered":"2023-07-14T18:23:11.226877+0000","last_clean":"2023-07-14T18:23:11.226877+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:23:11.226877+0000","last_undegraded":"2023-07-14T18:23:11.226877+0000","last_fullsized":"2023-07-14T18:23:11.226877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:59.578019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227969+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:23:11.227969+0000","last_peered":"2023-07-14T18:23:11.227969+0000","last_clean":"2023-07-14T18:23:11.227969+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:23:11.227969+0000","last_undegraded":"2023-07-14T18:23:11.227969+0000","last_fullsized":"2023-07-14T18:23:11.227969+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:23.742307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226855+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:23:11.226855+0000","last_peered":"2023-07-14T18:23:11.226855+0000","last_clean":"2023-07-14T18:23:11.226855+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:23:11.226855+0000","last_undegraded":"2023-07-14T18:23:11.226855+0000","last_fullsized":"2023-07-14T18:23:11.226855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:13.892292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227338+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:23:11.227338+0000","last_peered":"2023-07-14T18:23:11.227338+0000","last_clean":"2023-07-14T18:23:11.227338+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:23:11.227338+0000","last_undegraded":"2023-07-14T18:23:11.227338+0000","last_fullsized":"2023-07-14T18:23:11.227338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:36:55.710853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.285894+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:23:28.285894+0000","last_peered":"2023-07-14T18:23:28.285894+0000","last_clean":"2023-07-14T18:23:28.285894+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:23:28.285894+0000","last_undegraded":"2023-07-14T18:23:28.285894+0000","last_fullsized":"2023-07-14T18:23:28.285894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:30.855601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447648+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:23:10.447648+0000","last_peered":"2023-07-14T18:23:10.447648+0000","last_clean":"2023-07-14T18:23:10.447648+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:23:10.447648+0000","last_undegraded":"2023-07-14T18:23:10.447648+0000","last_fullsized":"2023-07-14T18:23:10.447648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:35.121798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227220+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:23:11.227220+0000","last_peered":"2023-07-14T18:23:11.227220+0000","last_clean":"2023-07-14T18:23:11.227220+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:23:11.227220+0000","last_undegraded":"2023-07-14T18:23:11.227220+0000","last_fullsized":"2023-07-14T18:23:11.227220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:10.747246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270798+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:23:27.270798+0000","last_peered":"2023-07-14T18:23:27.270798+0000","last_clean":"2023-07-14T18:23:27.270798+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:23:27.270798+0000","last_undegraded":"2023-07-14T18:23:27.270798+0000","last_fullsized":"2023-07-14T18:23:27.270798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:38.727870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226921+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:23:11.226921+0000","last_peered":"2023-07-14T18:23:11.226921+0000","last_clean":"2023-07-14T18:23:11.226921+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:23:11.226921+0000","last_undegraded":"2023-07-14T18:23:11.226921+0000","last_fullsized":"2023-07-14T18:23:11.226921+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:28:05.734032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445429+0000","last_change":"2023-07-14T18:23:09.448538+0000","last_active":"2023-07-14T18:23:10.445429+0000","last_peered":"2023-07-14T18:23:10.445429+0000","last_clean":"2023-07-14T18:23:10.445429+0000","last_became_active":"2023-07-14T18:23:09.448108+0000","last_became_peered":"2023-07-14T18:23:09.448108+0000","last_unstale":"2023-07-14T18:23:10.445429+0000","last_undegraded":"2023-07-14T18:23:10.445429+0000","last_fullsized":"2023-07-14T18:23:10.445429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:11.225500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:01.363114+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:24:01.363114+0000","last_peered":"2023-07-14T18:24:01.363114+0000","last_clean":"2023-07-14T18:24:01.363114+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:24:01.363114+0000","last_undegraded":"2023-07-14T18:24:01.363114+0000","last_fullsized":"2023-07-14T18:24:01.363114+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:27:13.492386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226906+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:23:11.226906+0000","last_peered":"2023-07-14T18:23:11.226906+0000","last_clean":"2023-07-14T18:23:11.226906+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:23:11.226906+0000","last_undegraded":"2023-07-14T18:23:11.226906+0000","last_fullsized":"2023-07-14T18:23:11.226906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:51:56.191875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445464+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:23:10.445464+0000","last_peered":"2023-07-14T18:23:10.445464+0000","last_clean":"2023-07-14T18:23:10.445464+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:23:10.445464+0000","last_undegraded":"2023-07-14T18:23:10.445464+0000","last_fullsized":"2023-07-14T18:23:10.445464+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:14.943827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263216+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:23:26.263216+0000","last_peered":"2023-07-14T18:23:26.263216+0000","last_clean":"2023-07-14T18:23:26.263216+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:23:26.263216+0000","last_undegraded":"2023-07-14T18:23:26.263216+0000","last_fullsized":"2023-07-14T18:23:26.263216+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:41.156493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229469+0000","last_change":"2023-07-14T18:23:05.254706+0000","last_active":"2023-07-14T18:23:11.229469+0000","last_peered":"2023-07-14T18:23:11.229469+0000","last_clean":"2023-07-14T18:23:11.229469+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.229469+0000","last_undegraded":"2023-07-14T18:23:11.229469+0000","last_fullsized":"2023-07-14T18:23:11.229469+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:06:59.136778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227564+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:23:11.227564+0000","last_peered":"2023-07-14T18:23:11.227564+0000","last_clean":"2023-07-14T18:23:11.227564+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:23:11.227564+0000","last_undegraded":"2023-07-14T18:23:11.227564+0000","last_fullsized":"2023-07-14T18:23:11.227564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:46:44.645790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.254386+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:23:25.254386+0000","last_peered":"2023-07-14T18:23:25.254386+0000","last_clean":"2023-07-14T18:23:25.254386+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:23:25.254386+0000","last_undegraded":"2023-07-14T18:23:25.254386+0000","last_fullsized":"2023-07-14T18:23:25.254386+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:35.962587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226991+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:23:11.226991+0000","last_peered":"2023-07-14T18:23:11.226991+0000","last_clean":"2023-07-14T18:23:11.226991+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:23:11.226991+0000","last_undegraded":"2023-07-14T18:23:11.226991+0000","last_fullsized":"2023-07-14T18:23:11.226991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:17.102026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226340+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:23:11.226340+0000","last_peered":"2023-07-14T18:23:11.226340+0000","last_clean":"2023-07-14T18:23:11.226340+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:23:11.226340+0000","last_undegraded":"2023-07-14T18:23:11.226340+0000","last_fullsized":"2023-07-14T18:23:11.226340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:06.464493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.269047+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:23:27.269047+0000","last_peered":"2023-07-14T18:23:27.269047+0000","last_clean":"2023-07-14T18:23:27.269047+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:23:27.269047+0000","last_undegraded":"2023-07-14T18:23:27.269047+0000","last_fullsized":"2023-07-14T18:23:27.269047+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:30:28.512912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444104+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:23:10.444104+0000","last_peered":"2023-07-14T18:23:10.444104+0000","last_clean":"2023-07-14T18:23:10.444104+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:23:10.444104+0000","last_undegraded":"2023-07-14T18:23:10.444104+0000","last_fullsized":"2023-07-14T18:23:10.444104+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:14.783284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227144+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:23:11.227144+0000","last_peered":"2023-07-14T18:23:11.227144+0000","last_clean":"2023-07-14T18:23:11.227144+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:23:11.227144+0000","last_undegraded":"2023-07-14T18:23:11.227144+0000","last_fullsized":"2023-07-14T18:23:11.227144+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:11:45.259664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":94,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:01.343509+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:24:01.343509+0000","last_peered":"2023-07-14T18:24:01.343509+0000","last_clean":"2023-07-14T18:24:01.343509+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:24:01.343509+0000","last_undegraded":"2023-07-14T18:24:01.343509+0000","last_fullsized":"2023-07-14T18:24:01.343509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:16:59.026085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447741+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:23:10.447741+0000","last_peered":"2023-07-14T18:23:10.447741+0000","last_clean":"2023-07-14T18:23:10.447741+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:23:10.447741+0000","last_undegraded":"2023-07-14T18:23:10.447741+0000","last_fullsized":"2023-07-14T18:23:10.447741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:46.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.443969+0000","last_change":"2023-07-14T18:23:09.450239+0000","last_active":"2023-07-14T18:23:10.443969+0000","last_peered":"2023-07-14T18:23:10.443969+0000","last_clean":"2023-07-14T18:23:10.443969+0000","last_became_active":"2023-07-14T18:23:09.449827+0000","last_became_peered":"2023-07-14T18:23:09.449827+0000","last_unstale":"2023-07-14T18:23:10.443969+0000","last_undegraded":"2023-07-14T18:23:10.443969+0000","last_fullsized":"2023-07-14T18:23:10.443969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:45:48.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270511+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:23:27.270511+0000","last_peered":"2023-07-14T18:23:27.270511+0000","last_clean":"2023-07-14T18:23:27.270511+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:23:27.270511+0000","last_undegraded":"2023-07-14T18:23:27.270511+0000","last_fullsized":"2023-07-14T18:23:27.270511+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:17:16.477875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226974+0000","last_change":"2023-07-14T18:23:05.255321+0000","last_active":"2023-07-14T18:23:11.226974+0000","last_peered":"2023-07-14T18:23:11.226974+0000","last_clean":"2023-07-14T18:23:11.226974+0000","last_became_active":"2023-07-14T18:23:05.254443+0000","last_became_peered":"2023-07-14T18:23:05.254443+0000","last_unstale":"2023-07-14T18:23:11.226974+0000","last_undegraded":"2023-07-14T18:23:11.226974+0000","last_fullsized":"2023-07-14T18:23:11.226974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:47:37.938549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227625+0000","last_change":"2023-07-14T18:23:09.448642+0000","last_active":"2023-07-14T18:23:11.227625+0000","last_peered":"2023-07-14T18:23:11.227625+0000","last_clean":"2023-07-14T18:23:11.227625+0000","last_became_active":"2023-07-14T18:23:09.448079+0000","last_became_peered":"2023-07-14T18:23:09.448079+0000","last_unstale":"2023-07-14T18:23:11.227625+0000","last_undegraded":"2023-07-14T18:23:11.227625+0000","last_fullsized":"2023-07-14T18:23:11.227625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:25:39.875566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:26.263528+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:23:26.263528+0000","last_peered":"2023-07-14T18:23:26.263528+0000","last_clean":"2023-07-14T18:23:26.263528+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:23:26.263528+0000","last_undegraded":"2023-07-14T18:23:26.263528+0000","last_fullsized":"2023-07-14T18:23:26.263528+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:06:35.880799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447624+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:23:10.447624+0000","last_peered":"2023-07-14T18:23:10.447624+0000","last_clean":"2023-07-14T18:23:10.447624+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:23:10.447624+0000","last_undegraded":"2023-07-14T18:23:10.447624+0000","last_fullsized":"2023-07-14T18:23:10.447624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:01.328580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444552+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:23:10.444552+0000","last_peered":"2023-07-14T18:23:10.444552+0000","last_clean":"2023-07-14T18:23:10.444552+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:23:10.444552+0000","last_undegraded":"2023-07-14T18:23:10.444552+0000","last_fullsized":"2023-07-14T18:23:10.444552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:07:35.036816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'33","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:01.343517+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:24:01.343517+0000","last_peered":"2023-07-14T18:24:01.343517+0000","last_clean":"2023-07-14T18:24:01.343517+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:24:01.343517+0000","last_undegraded":"2023-07-14T18:24:01.343517+0000","last_fullsized":"2023-07-14T18:24:01.343517+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:29:45.329055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":74,"num_read_kb":93,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228668+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:23:11.228668+0000","last_peered":"2023-07-14T18:23:11.228668+0000","last_clean":"2023-07-14T18:23:11.228668+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:23:11.228668+0000","last_undegraded":"2023-07-14T18:23:11.228668+0000","last_fullsized":"2023-07-14T18:23:11.228668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:23.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227342+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:23:11.227342+0000","last_peered":"2023-07-14T18:23:11.227342+0000","last_clean":"2023-07-14T18:23:11.227342+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:23:11.227342+0000","last_undegraded":"2023-07-14T18:23:11.227342+0000","last_fullsized":"2023-07-14T18:23:11.227342+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:33:09.226720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.284411+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:23:28.284411+0000","last_peered":"2023-07-14T18:23:28.284411+0000","last_clean":"2023-07-14T18:23:28.284411+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:23:28.284411+0000","last_undegraded":"2023-07-14T18:23:28.284411+0000","last_fullsized":"2023-07-14T18:23:28.284411+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:45:15.877531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444561+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:23:10.444561+0000","last_peered":"2023-07-14T18:23:10.444561+0000","last_clean":"2023-07-14T18:23:10.444561+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:23:10.444561+0000","last_undegraded":"2023-07-14T18:23:10.444561+0000","last_fullsized":"2023-07-14T18:23:10.444561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:11:50.717670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226859+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:23:11.226859+0000","last_peered":"2023-07-14T18:23:11.226859+0000","last_clean":"2023-07-14T18:23:11.226859+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:23:11.226859+0000","last_undegraded":"2023-07-14T18:23:11.226859+0000","last_fullsized":"2023-07-14T18:23:11.226859+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:40:53.808997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":28,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.269446+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:23:27.269446+0000","last_peered":"2023-07-14T18:23:27.269446+0000","last_clean":"2023-07-14T18:23:27.269446+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:23:27.269446+0000","last_undegraded":"2023-07-14T18:23:27.269446+0000","last_fullsized":"2023-07-14T18:23:27.269446+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:25:36.450416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610444+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:23:10.610444+0000","last_peered":"2023-07-14T18:23:10.610444+0000","last_clean":"2023-07-14T18:23:10.610444+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:23:10.610444+0000","last_undegraded":"2023-07-14T18:23:10.610444+0000","last_fullsized":"2023-07-14T18:23:10.610444+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:21:21.561860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444636+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:23:10.444636+0000","last_peered":"2023-07-14T18:23:10.444636+0000","last_clean":"2023-07-14T18:23:10.444636+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:23:10.444636+0000","last_undegraded":"2023-07-14T18:23:10.444636+0000","last_fullsized":"2023-07-14T18:23:10.444636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:41.687245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:03.350262+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:24:03.350262+0000","last_peered":"2023-07-14T18:24:03.350262+0000","last_clean":"2023-07-14T18:24:03.350262+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:24:03.350262+0000","last_undegraded":"2023-07-14T18:24:03.350262+0000","last_fullsized":"2023-07-14T18:24:03.350262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:54:03.971193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610150+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:23:10.610150+0000","last_peered":"2023-07-14T18:23:10.610150+0000","last_clean":"2023-07-14T18:23:10.610150+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:23:10.610150+0000","last_undegraded":"2023-07-14T18:23:10.610150+0000","last_fullsized":"2023-07-14T18:23:10.610150+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:26.204125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445599+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:23:10.445599+0000","last_peered":"2023-07-14T18:23:10.445599+0000","last_clean":"2023-07-14T18:23:10.445599+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:23:10.445599+0000","last_undegraded":"2023-07-14T18:23:10.445599+0000","last_fullsized":"2023-07-14T18:23:10.445599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:32:00.074176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270941+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:23:27.270941+0000","last_peered":"2023-07-14T18:23:27.270941+0000","last_clean":"2023-07-14T18:23:27.270941+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:23:27.270941+0000","last_undegraded":"2023-07-14T18:23:27.270941+0000","last_fullsized":"2023-07-14T18:23:27.270941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:45.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228441+0000","last_change":"2023-07-14T18:23:05.255683+0000","last_active":"2023-07-14T18:23:11.228441+0000","last_peered":"2023-07-14T18:23:11.228441+0000","last_clean":"2023-07-14T18:23:11.228441+0000","last_became_active":"2023-07-14T18:23:05.255360+0000","last_became_peered":"2023-07-14T18:23:05.255360+0000","last_unstale":"2023-07-14T18:23:11.228441+0000","last_undegraded":"2023-07-14T18:23:11.228441+0000","last_fullsized":"2023-07-14T18:23:11.228441+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:23.880165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228337+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:23:11.228337+0000","last_peered":"2023-07-14T18:23:11.228337+0000","last_clean":"2023-07-14T18:23:11.228337+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:23:11.228337+0000","last_undegraded":"2023-07-14T18:23:11.228337+0000","last_fullsized":"2023-07-14T18:23:11.228337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:18.472121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.284458+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:23:28.284458+0000","last_peered":"2023-07-14T18:23:28.284458+0000","last_clean":"2023-07-14T18:23:28.284458+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:23:28.284458+0000","last_undegraded":"2023-07-14T18:23:28.284458+0000","last_fullsized":"2023-07-14T18:23:28.284458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:33.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227454+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:23:11.227454+0000","last_peered":"2023-07-14T18:23:11.227454+0000","last_clean":"2023-07-14T18:23:11.227454+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:23:11.227454+0000","last_undegraded":"2023-07-14T18:23:11.227454+0000","last_fullsized":"2023-07-14T18:23:11.227454+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:06:55.074433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226809+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:23:11.226809+0000","last_peered":"2023-07-14T18:23:11.226809+0000","last_clean":"2023-07-14T18:23:11.226809+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:23:11.226809+0000","last_undegraded":"2023-07-14T18:23:11.226809+0000","last_fullsized":"2023-07-14T18:23:11.226809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:44.909597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444437+0000","last_change":"2023-07-14T18:23:04.418035+0000","last_active":"2023-07-14T18:23:10.444437+0000","last_peered":"2023-07-14T18:23:10.444437+0000","last_clean":"2023-07-14T18:23:10.444437+0000","last_became_active":"2023-07-14T18:23:04.417645+0000","last_became_peered":"2023-07-14T18:23:04.417645+0000","last_unstale":"2023-07-14T18:23:10.444437+0000","last_undegraded":"2023-07-14T18:23:10.444437+0000","last_fullsized":"2023-07-14T18:23:10.444437+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:08.545074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227920+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:23:11.227920+0000","last_peered":"2023-07-14T18:23:11.227920+0000","last_clean":"2023-07-14T18:23:11.227920+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:23:11.227920+0000","last_undegraded":"2023-07-14T18:23:11.227920+0000","last_fullsized":"2023-07-14T18:23:11.227920+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:47.382928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445312+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:23:10.445312+0000","last_peered":"2023-07-14T18:23:10.445312+0000","last_clean":"2023-07-14T18:23:10.445312+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:23:10.445312+0000","last_undegraded":"2023-07-14T18:23:10.445312+0000","last_fullsized":"2023-07-14T18:23:10.445312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:36:47.920949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:02.349061+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:24:02.349061+0000","last_peered":"2023-07-14T18:24:02.349061+0000","last_clean":"2023-07-14T18:24:02.349061+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:24:02.349061+0000","last_undegraded":"2023-07-14T18:24:02.349061+0000","last_fullsized":"2023-07-14T18:24:02.349061+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:22:04.367333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227207+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:23:11.227207+0000","last_peered":"2023-07-14T18:23:11.227207+0000","last_clean":"2023-07-14T18:23:11.227207+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:23:11.227207+0000","last_undegraded":"2023-07-14T18:23:11.227207+0000","last_fullsized":"2023-07-14T18:23:11.227207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:56:42.916396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445652+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:23:10.445652+0000","last_peered":"2023-07-14T18:23:10.445652+0000","last_clean":"2023-07-14T18:23:10.445652+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:23:10.445652+0000","last_undegraded":"2023-07-14T18:23:10.445652+0000","last_fullsized":"2023-07-14T18:23:10.445652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:51:24.527443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228927+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:23:11.228927+0000","last_peered":"2023-07-14T18:23:11.228927+0000","last_clean":"2023-07-14T18:23:11.228927+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:23:11.228927+0000","last_undegraded":"2023-07-14T18:23:11.228927+0000","last_fullsized":"2023-07-14T18:23:11.228927+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:09:15.545609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":65,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:03.349502+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:24:03.349502+0000","last_peered":"2023-07-14T18:24:03.349502+0000","last_clean":"2023-07-14T18:24:03.349502+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:24:03.349502+0000","last_undegraded":"2023-07-14T18:24:03.349502+0000","last_fullsized":"2023-07-14T18:24:03.349502+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:33:45.754899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610640+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:23:10.610640+0000","last_peered":"2023-07-14T18:23:10.610640+0000","last_clean":"2023-07-14T18:23:10.610640+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:23:10.610640+0000","last_undegraded":"2023-07-14T18:23:10.610640+0000","last_fullsized":"2023-07-14T18:23:10.610640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:17.385324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226985+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:23:11.226985+0000","last_peered":"2023-07-14T18:23:11.226985+0000","last_clean":"2023-07-14T18:23:11.226985+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:23:11.226985+0000","last_undegraded":"2023-07-14T18:23:11.226985+0000","last_fullsized":"2023-07-14T18:23:11.226985+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:48.279521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.286159+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:23:28.286159+0000","last_peered":"2023-07-14T18:23:28.286159+0000","last_clean":"2023-07-14T18:23:28.286159+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:23:28.286159+0000","last_undegraded":"2023-07-14T18:23:28.286159+0000","last_fullsized":"2023-07-14T18:23:28.286159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:52.919560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":344,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.609987+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:23:10.609987+0000","last_peered":"2023-07-14T18:23:10.609987+0000","last_clean":"2023-07-14T18:23:10.609987+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:23:10.609987+0000","last_undegraded":"2023-07-14T18:23:10.609987+0000","last_fullsized":"2023-07-14T18:23:10.609987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:43.598474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227359+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:23:11.227359+0000","last_peered":"2023-07-14T18:23:11.227359+0000","last_clean":"2023-07-14T18:23:11.227359+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:23:11.227359+0000","last_undegraded":"2023-07-14T18:23:11.227359+0000","last_fullsized":"2023-07-14T18:23:11.227359+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:34:29.533657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444959+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:23:10.444959+0000","last_peered":"2023-07-14T18:23:10.444959+0000","last_clean":"2023-07-14T18:23:10.444959+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:23:10.444959+0000","last_undegraded":"2023-07-14T18:23:10.444959+0000","last_fullsized":"2023-07-14T18:23:10.444959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:08.456204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:25.252048+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:23:25.252048+0000","last_peered":"2023-07-14T18:23:25.252048+0000","last_clean":"2023-07-14T18:23:25.252048+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:23:25.252048+0000","last_undegraded":"2023-07-14T18:23:25.252048+0000","last_fullsized":"2023-07-14T18:23:25.252048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:57.271343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226827+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:23:11.226827+0000","last_peered":"2023-07-14T18:23:11.226827+0000","last_clean":"2023-07-14T18:23:11.226827+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:23:11.226827+0000","last_undegraded":"2023-07-14T18:23:11.226827+0000","last_fullsized":"2023-07-14T18:23:11.226827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:14:41.538513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270466+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:23:27.270466+0000","last_peered":"2023-07-14T18:23:27.270466+0000","last_clean":"2023-07-14T18:23:27.270466+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:23:27.270466+0000","last_undegraded":"2023-07-14T18:23:27.270466+0000","last_fullsized":"2023-07-14T18:23:27.270466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:29.140961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445507+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:23:10.445507+0000","last_peered":"2023-07-14T18:23:10.445507+0000","last_clean":"2023-07-14T18:23:10.445507+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:23:10.445507+0000","last_undegraded":"2023-07-14T18:23:10.445507+0000","last_fullsized":"2023-07-14T18:23:10.445507+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:58:01.714171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226933+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:23:11.226933+0000","last_peered":"2023-07-14T18:23:11.226933+0000","last_clean":"2023-07-14T18:23:11.226933+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:23:11.226933+0000","last_undegraded":"2023-07-14T18:23:11.226933+0000","last_fullsized":"2023-07-14T18:23:11.226933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:06:06.174345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:29.315427+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:23:29.315427+0000","last_peered":"2023-07-14T18:23:29.315427+0000","last_clean":"2023-07-14T18:23:29.315427+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:23:29.315427+0000","last_undegraded":"2023-07-14T18:23:29.315427+0000","last_fullsized":"2023-07-14T18:23:29.315427+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:48:38.063154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227450+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:23:11.227450+0000","last_peered":"2023-07-14T18:23:11.227450+0000","last_clean":"2023-07-14T18:23:11.227450+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:23:11.227450+0000","last_undegraded":"2023-07-14T18:23:11.227450+0000","last_fullsized":"2023-07-14T18:23:11.227450+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:39.395014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445040+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:23:10.445040+0000","last_peered":"2023-07-14T18:23:10.445040+0000","last_clean":"2023-07-14T18:23:10.445040+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:23:10.445040+0000","last_undegraded":"2023-07-14T18:23:10.445040+0000","last_fullsized":"2023-07-14T18:23:10.445040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:10:54.870365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270819+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:23:27.270819+0000","last_peered":"2023-07-14T18:23:27.270819+0000","last_clean":"2023-07-14T18:23:27.270819+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:23:27.270819+0000","last_undegraded":"2023-07-14T18:23:27.270819+0000","last_fullsized":"2023-07-14T18:23:27.270819+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:54:38.614796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226909+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:23:11.226909+0000","last_peered":"2023-07-14T18:23:11.226909+0000","last_clean":"2023-07-14T18:23:11.226909+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:23:11.226909+0000","last_undegraded":"2023-07-14T18:23:11.226909+0000","last_fullsized":"2023-07-14T18:23:11.226909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:43.634763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227411+0000","last_change":"2023-07-14T18:23:09.449832+0000","last_active":"2023-07-14T18:23:11.227411+0000","last_peered":"2023-07-14T18:23:11.227411+0000","last_clean":"2023-07-14T18:23:11.227411+0000","last_became_active":"2023-07-14T18:23:09.448997+0000","last_became_peered":"2023-07-14T18:23:09.448997+0000","last_unstale":"2023-07-14T18:23:11.227411+0000","last_undegraded":"2023-07-14T18:23:11.227411+0000","last_fullsized":"2023-07-14T18:23:11.227411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:56.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242161+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:23:24.242161+0000","last_peered":"2023-07-14T18:23:24.242161+0000","last_clean":"2023-07-14T18:23:24.242161+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:23:24.242161+0000","last_undegraded":"2023-07-14T18:23:24.242161+0000","last_fullsized":"2023-07-14T18:23:24.242161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:46.321986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226403+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:23:11.226403+0000","last_peered":"2023-07-14T18:23:11.226403+0000","last_clean":"2023-07-14T18:23:11.226403+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:23:11.226403+0000","last_undegraded":"2023-07-14T18:23:11.226403+0000","last_fullsized":"2023-07-14T18:23:11.226403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:03:27.653641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444903+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:23:10.444903+0000","last_peered":"2023-07-14T18:23:10.444903+0000","last_clean":"2023-07-14T18:23:10.444903+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:23:10.444903+0000","last_undegraded":"2023-07-14T18:23:10.444903+0000","last_fullsized":"2023-07-14T18:23:10.444903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:24:40.273437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268661+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:23:27.268661+0000","last_peered":"2023-07-14T18:23:27.268661+0000","last_clean":"2023-07-14T18:23:27.268661+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:23:27.268661+0000","last_undegraded":"2023-07-14T18:23:27.268661+0000","last_fullsized":"2023-07-14T18:23:27.268661+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:15:25.573588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229498+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:23:11.229498+0000","last_peered":"2023-07-14T18:23:11.229498+0000","last_clean":"2023-07-14T18:23:11.229498+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:23:11.229498+0000","last_undegraded":"2023-07-14T18:23:11.229498+0000","last_fullsized":"2023-07-14T18:23:11.229498+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:56:55.294012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227722+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:23:11.227722+0000","last_peered":"2023-07-14T18:23:11.227722+0000","last_clean":"2023-07-14T18:23:11.227722+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:23:11.227722+0000","last_undegraded":"2023-07-14T18:23:11.227722+0000","last_fullsized":"2023-07-14T18:23:11.227722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:26:45.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.600831+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:23:27.600831+0000","last_peered":"2023-07-14T18:23:27.600831+0000","last_clean":"2023-07-14T18:23:27.600831+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:23:27.600831+0000","last_undegraded":"2023-07-14T18:23:27.600831+0000","last_fullsized":"2023-07-14T18:23:27.600831+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:57:27.449607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227548+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:23:11.227548+0000","last_peered":"2023-07-14T18:23:11.227548+0000","last_clean":"2023-07-14T18:23:11.227548+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:23:11.227548+0000","last_undegraded":"2023-07-14T18:23:11.227548+0000","last_fullsized":"2023-07-14T18:23:11.227548+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:58.508966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227371+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:23:11.227371+0000","last_peered":"2023-07-14T18:23:11.227371+0000","last_clean":"2023-07-14T18:23:11.227371+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:23:11.227371+0000","last_undegraded":"2023-07-14T18:23:11.227371+0000","last_fullsized":"2023-07-14T18:23:11.227371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:20.402932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242000+0000","last_change":"2023-07-14T18:23:04.424951+0000","last_active":"2023-07-14T18:23:24.242000+0000","last_peered":"2023-07-14T18:23:24.242000+0000","last_clean":"2023-07-14T18:23:24.242000+0000","last_became_active":"2023-07-14T18:23:04.424641+0000","last_became_peered":"2023-07-14T18:23:04.424641+0000","last_unstale":"2023-07-14T18:23:24.242000+0000","last_undegraded":"2023-07-14T18:23:24.242000+0000","last_fullsized":"2023-07-14T18:23:24.242000+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:06:25.476705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610293+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:23:10.610293+0000","last_peered":"2023-07-14T18:23:10.610293+0000","last_clean":"2023-07-14T18:23:10.610293+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:23:10.610293+0000","last_undegraded":"2023-07-14T18:23:10.610293+0000","last_fullsized":"2023-07-14T18:23:10.610293+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:11.675787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447265+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:23:10.447265+0000","last_peered":"2023-07-14T18:23:10.447265+0000","last_clean":"2023-07-14T18:23:10.447265+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:23:10.447265+0000","last_undegraded":"2023-07-14T18:23:10.447265+0000","last_fullsized":"2023-07-14T18:23:10.447265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:43.737345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:01.344022+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:24:01.344022+0000","last_peered":"2023-07-14T18:24:01.344022+0000","last_clean":"2023-07-14T18:24:01.344022+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:24:01.344022+0000","last_undegraded":"2023-07-14T18:24:01.344022+0000","last_fullsized":"2023-07-14T18:24:01.344022+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:58:04.272902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":37,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610785+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:23:10.610785+0000","last_peered":"2023-07-14T18:23:10.610785+0000","last_clean":"2023-07-14T18:23:10.610785+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:23:10.610785+0000","last_undegraded":"2023-07-14T18:23:10.610785+0000","last_fullsized":"2023-07-14T18:23:10.610785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:27:06.003284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226097+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:23:11.226097+0000","last_peered":"2023-07-14T18:23:11.226097+0000","last_clean":"2023-07-14T18:23:11.226097+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:23:11.226097+0000","last_undegraded":"2023-07-14T18:23:11.226097+0000","last_fullsized":"2023-07-14T18:23:11.226097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:26.374670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268237+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:23:27.268237+0000","last_peered":"2023-07-14T18:23:27.268237+0000","last_clean":"2023-07-14T18:23:27.268237+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:23:27.268237+0000","last_undegraded":"2023-07-14T18:23:27.268237+0000","last_fullsized":"2023-07-14T18:23:27.268237+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:05:36.820471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.610456+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:23:10.610456+0000","last_peered":"2023-07-14T18:23:10.610456+0000","last_clean":"2023-07-14T18:23:10.610456+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:23:10.610456+0000","last_undegraded":"2023-07-14T18:23:10.610456+0000","last_fullsized":"2023-07-14T18:23:10.610456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:55:16.680648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228516+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:23:11.228516+0000","last_peered":"2023-07-14T18:23:11.228516+0000","last_clean":"2023-07-14T18:23:11.228516+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:23:11.228516+0000","last_undegraded":"2023-07-14T18:23:11.228516+0000","last_fullsized":"2023-07-14T18:23:11.228516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:14:45.616005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.268218+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:23:27.268218+0000","last_peered":"2023-07-14T18:23:27.268218+0000","last_clean":"2023-07-14T18:23:27.268218+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:23:27.268218+0000","last_undegraded":"2023-07-14T18:23:27.268218+0000","last_fullsized":"2023-07-14T18:23:27.268218+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:27:35.670294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.611027+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:23:10.611027+0000","last_peered":"2023-07-14T18:23:10.611027+0000","last_clean":"2023-07-14T18:23:10.611027+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:23:10.611027+0000","last_undegraded":"2023-07-14T18:23:10.611027+0000","last_fullsized":"2023-07-14T18:23:10.611027+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:15:51.422332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.229684+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:23:11.229684+0000","last_peered":"2023-07-14T18:23:11.229684+0000","last_clean":"2023-07-14T18:23:11.229684+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:23:11.229684+0000","last_undegraded":"2023-07-14T18:23:11.229684+0000","last_fullsized":"2023-07-14T18:23:11.229684+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:37:17.955330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445263+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:23:10.445263+0000","last_peered":"2023-07-14T18:23:10.445263+0000","last_clean":"2023-07-14T18:23:10.445263+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:23:10.445263+0000","last_undegraded":"2023-07-14T18:23:10.445263+0000","last_fullsized":"2023-07-14T18:23:10.445263+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:47.687598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445138+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:23:10.445138+0000","last_peered":"2023-07-14T18:23:10.445138+0000","last_clean":"2023-07-14T18:23:10.445138+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:23:10.445138+0000","last_undegraded":"2023-07-14T18:23:10.445138+0000","last_fullsized":"2023-07-14T18:23:10.445138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:39:22.609183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226339+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:23:11.226339+0000","last_peered":"2023-07-14T18:23:11.226339+0000","last_clean":"2023-07-14T18:23:11.226339+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:23:11.226339+0000","last_undegraded":"2023-07-14T18:23:11.226339+0000","last_fullsized":"2023-07-14T18:23:11.226339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:45.583600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445397+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:23:10.445397+0000","last_peered":"2023-07-14T18:23:10.445397+0000","last_clean":"2023-07-14T18:23:10.445397+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:23:10.445397+0000","last_undegraded":"2023-07-14T18:23:10.445397+0000","last_fullsized":"2023-07-14T18:23:10.445397+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:10.666041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227616+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:23:11.227616+0000","last_peered":"2023-07-14T18:23:11.227616+0000","last_clean":"2023-07-14T18:23:11.227616+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:23:11.227616+0000","last_undegraded":"2023-07-14T18:23:11.227616+0000","last_fullsized":"2023-07-14T18:23:11.227616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:41:50.867762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.445702+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:23:10.445702+0000","last_peered":"2023-07-14T18:23:10.445702+0000","last_clean":"2023-07-14T18:23:10.445702+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:23:10.445702+0000","last_undegraded":"2023-07-14T18:23:10.445702+0000","last_fullsized":"2023-07-14T18:23:10.445702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:50.705252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227634+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:23:11.227634+0000","last_peered":"2023-07-14T18:23:11.227634+0000","last_clean":"2023-07-14T18:23:11.227634+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:23:11.227634+0000","last_undegraded":"2023-07-14T18:23:11.227634+0000","last_fullsized":"2023-07-14T18:23:11.227634+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:59:57.010106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228805+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:23:11.228805+0000","last_peered":"2023-07-14T18:23:11.228805+0000","last_clean":"2023-07-14T18:23:11.228805+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:23:11.228805+0000","last_undegraded":"2023-07-14T18:23:11.228805+0000","last_fullsized":"2023-07-14T18:23:11.228805+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:27.226698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227229+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:23:11.227229+0000","last_peered":"2023-07-14T18:23:11.227229+0000","last_clean":"2023-07-14T18:23:11.227229+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:23:11.227229+0000","last_undegraded":"2023-07-14T18:23:11.227229+0000","last_fullsized":"2023-07-14T18:23:11.227229+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:21:50.246761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447443+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:23:10.447443+0000","last_peered":"2023-07-14T18:23:10.447443+0000","last_clean":"2023-07-14T18:23:10.447443+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:23:10.447443+0000","last_undegraded":"2023-07-14T18:23:10.447443+0000","last_fullsized":"2023-07-14T18:23:10.447443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:22.891172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.444900+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:23:10.444900+0000","last_peered":"2023-07-14T18:23:10.444900+0000","last_clean":"2023-07-14T18:23:10.444900+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:23:10.444900+0000","last_undegraded":"2023-07-14T18:23:10.444900+0000","last_fullsized":"2023-07-14T18:23:10.444900+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:52.929194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227023+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:23:11.227023+0000","last_peered":"2023-07-14T18:23:11.227023+0000","last_clean":"2023-07-14T18:23:11.227023+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:23:11.227023+0000","last_undegraded":"2023-07-14T18:23:11.227023+0000","last_fullsized":"2023-07-14T18:23:11.227023+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:59:19.793596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":387,"num_read_kb":263,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514519,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":458554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57999999999999996}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.67000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77800000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.89200000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.84399999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93999999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.92700000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.94999999999999996}]}]},{"osd":10,"up_from":69,"seq":296352743451,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1103531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.52500000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58299999999999996}]},{"osd":4,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.51500000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.745}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65900000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.256,"5min":1.256,"15min":1.256},"last":0.56899999999999995}]},{"osd":11,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.76000000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906976,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.499,"15min":0.46400000000000002},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":5.8559999999999999},"last":0.442},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.308}]},{"osd":1,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.874,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.5140000000000002,"5min":5.5140000000000002,"15min":5.5140000000000002},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.71299999999999997,"15min":0.68799999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.641,"5min":5.641,"15min":5.641},"last":0.68700000000000006}]},{"osd":3,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.54800000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5679999999999996,"5min":5.5679999999999996,"15min":5.5679999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.46500000000000002}]},{"osd":5,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78200000000000003,"15min":0.79400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.70799999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80400000000000005,"15min":0.81399999999999995},"min":{"1min":0.53000000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.60799999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.85299999999999998,"15min":0.82899999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.875},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.59899999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:03 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":458520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52400000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.79900000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.75}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.77500000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.73499999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70399999999999996,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.55200000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.91700000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.872,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":7,"up_from":50,"seq":214748364840,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28540,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716612,"statfs":{"total":95995035648,"available":95965810688,"internally_reserved":0,"allocated":1560576,"data_stored":1054599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.40200000000000002,"15min":0.39500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.66800000000000004,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.34699999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.40500000000000003,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76500000000000001,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47899999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57599999999999996,"15min":0.57399999999999995},"min":{"1min":0.39500000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.879,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.38},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.61599999999999999,"15min":0.61899999999999999},"min":{"1min":0.45900000000000002,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.751,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.44}]},{"osd":4,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.52800000000000002,"15min":0.52400000000000002},"min":{"1min":0.33200000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.502,"15min":0.497},"min":{"1min":0.313,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.63}]},{"osd":5,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.625,"5min":0.61599999999999999,"15min":0.61399999999999999},"min":{"1min":0.45400000000000001,"5min":0.44,"15min":0.44},"max":{"1min":0.81799999999999995,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.51100000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92600000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"min":{"1min":0.53900000000000003,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":1.02,"15min":1.02},"last":0.68700000000000006}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69899999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97099999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.65500000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71099999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":453325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.316},{"interface":"front","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40500000000000003},"min":{"1min":0.28299999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.33500000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53700000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.38500000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93300000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.45800000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96399999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.71599999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.80100000000000005,"5min":1.099,"15min":1.099},"last":0.68200000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.66800000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.41499999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.70199999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":513179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.70899999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.53400000000000003},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.749}]},{"osd":4,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.42999999999999999},"min":{"1min":0.35799999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71199999999999997,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.48499999999999999,"15min":0.46200000000000002},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.76700000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68600000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.64000000000000001,"15min":0.61299999999999999},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93999999999999995,"5min":1.089,"15min":1.089},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.64000000000000001,"15min":0.61499999999999999},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.498,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0249999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.623,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88900000000000001,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.82099999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69499999999999995,"15min":0.67900000000000005},"min":{"1min":0.54600000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.012,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.80900000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":1.173,"15min":1.173},"last":0.93999999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.097,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.89900000000000002}]},{"osd":11,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.878},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.95999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":455171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56899999999999995,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81100000000000005,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.96599999999999997,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.629},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.432}]},{"osd":4,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.54700000000000004},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.503},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.61499999999999999,"15min":0.57099999999999995},"min":{"1min":0.498,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.48599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89800000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45100000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74299999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97399999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.78500000000000003},"min":{"1min":0.503,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.85499999999999998,"5min":1.036,"15min":1.036},"last":0.80500000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.442,"15min":0.442},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.65500000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91400000000000003,"5min":1.075,"15min":1.075},"last":0.82299999999999995}]},{"osd":11,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.72099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":514131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.79100000000000004}]},{"osd":3,"last update":"Fri Jul 14 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60399999999999998}]},{"osd":4,"last update":"Fri Jul 14 18:22:58 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.56499999999999995,"15min":0.54900000000000004},"min":{"1min":0.47799999999999998,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76400000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.57299999999999995,"15min":0.55200000000000005},"min":{"1min":0.505,"5min":0.38,"15min":0.38},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.97799999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.72199999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.54100000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.81100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.83099999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.72299999999999998,"15min":0.70499999999999996},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.91900000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.056},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.94499999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.036}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":453073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41199999999999998,"15min":0.374},"min":{"1min":0.29099999999999998,"5min":0.161,"15min":0.161},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.47799999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"last":0.44900000000000001}]},{"osd":1,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.629,"15min":0.57899999999999996},"min":{"1min":0.41799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"last":0.73399999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":8.1820000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.84899999999999998}]},{"osd":4,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.68200000000000005,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.2210000000000001,"5min":8.2210000000000001,"15min":8.2210000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.749}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":0.83299999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.68500000000000005,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.69199999999999995,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.58699999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.70199999999999996,"15min":0.69799999999999995},"min":{"1min":0.55200000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.125,"5min":1.22,"15min":1.22},"last":0.871}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":1.28,"5min":0.879,"15min":0.81200000000000006},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.65200000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.88300000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]}]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.40899999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89000000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.89300000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58699999999999997}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71099999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56000000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28548,"kb_used_data":1532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716604,"statfs":{"total":95995035648,"available":95965802496,"internally_reserved":0,"allocated":1568768,"data_stored":1050219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.53500000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.501}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.65200000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83399999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.79500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66300000000000003,"15min":0.65500000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.075,"5min":1.075,"15min":1.075},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68200000000000005,"15min":0.67600000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0289999999999999,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.77500000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.81000000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89600000000000002,"5min":1.087,"15min":1.087},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.80200000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.98799999999999999,"5min":1.206,"15min":1.206},"last":0.871}]},{"osd":10,"last update":"Fri Jul 14 18:23:04 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.2350000000000001,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:24:07.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:24:07.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: pgmap v285: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 170 B/s wr, 9 op/s 2023-07-14T18:24:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.4", "id": [2, 11]}]': finished 2023-07-14T18:24:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11]}]': finished 2023-07-14T18:24:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T18:24:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 7]}]': finished 2023-07-14T18:24:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 8]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [9, 11]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [5, 10]}]': finished 2023-07-14T18:24:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:07 smithi144 ceph-mon[98417]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T18:24:08.444 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:08.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:08 smithi064 ceph-mon[96720]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:08.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:08 smithi064 ceph-mon[96720]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:24:08.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:08 smithi105 ceph-mon[98013]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:08.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:08 smithi105 ceph-mon[98013]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:24:08.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:08 smithi144 ceph-mon[98417]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:08.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:08 smithi144 ceph-mon[98417]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T18:24:09.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 -- 172.21.15.64:0/9790083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a50103ee0 msgr2=0x7f8a5010ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:09.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/9790083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a50103ee0 0x7f8a5010ba60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8a4800b0d0 tx=0x7f8a4800b3e0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 -- 172.21.15.64:0/9790083 shutdown_connections 2023-07-14T18:24:09.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/9790083 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a500fee40 0x7f8a500ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/9790083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a50103ee0 0x7f8a5010ba60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/9790083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 0x7f8a501037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 -- 172.21.15.64:0/9790083 >> 172.21.15.64:0/9790083 conn(0x7f8a50075ca0 msgr2=0x7f8a500760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:09.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 -- 172.21.15.64:0/9790083 shutdown_connections 2023-07-14T18:24:09.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.579+0000 7f8a54d4b700 1 -- 172.21.15.64:0/9790083 wait complete. 2023-07-14T18:24:09.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 Processor -- start 2023-07-14T18:24:09.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 -- start start 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 0x7f8a50195200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a50103ee0 0x7f8a501985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50198bc0 con 0x7f8a500fee40 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50198d30 con 0x7f8a500ff7e0 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a54d4b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a50198ef0 con 0x7f8a50103ee0 2023-07-14T18:24:09.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a4e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a4e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47890/0 (socket says 172.21.15.64:47890) 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.580+0000 7f8a4e59c700 1 -- 172.21.15.64:0/2735983124 learned_addr learned my addr 172.21.15.64:0/2735983124 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4ed9d700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a50103ee0 0x7f8a501985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a47fff700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 0x7f8a50195200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 -- 172.21.15.64:0/2735983124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a50103ee0 msgr2=0x7f8a501985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a50103ee0 0x7f8a501985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 -- 172.21.15.64:0/2735983124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 msgr2=0x7f8a50195200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 0x7f8a50195200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 -- 172.21.15.64:0/2735983124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a48009d00 con 0x7f8a500fee40 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a4e59c700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000d950 tx=0x7f8a4000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4000f890 con 0x7f8a500fee40 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a501991d0 con 0x7f8a500fee40 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a4000fe80 con 0x7f8a500fee40 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4000e3b0 con 0x7f8a500fee40 2023-07-14T18:24:09.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.581+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a501a1b20 con 0x7f8a500fee40 2023-07-14T18:24:09.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.582+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5004e340 con 0x7f8a500fee40 2023-07-14T18:24:09.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.583+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8a40010460 con 0x7f8a500fee40 2023-07-14T18:24:09.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.584+0000 7f8a477fe700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 0x7f8a3c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:09.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.584+0000 7f8a47fff700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 0x7f8a3c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:09.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.584+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a400ccb20 con 0x7f8a500fee40 2023-07-14T18:24:09.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.585+0000 7f8a47fff700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 0x7f8a3c0a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8a501962a0 tx=0x7f8a38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:09.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.587+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a40095e00 con 0x7f8a500fee40 2023-07-14T18:24:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:09 smithi105 ceph-mon[98013]: pgmap v288: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2023-07-14T18:24:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:09 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:09 smithi064 ceph-mon[96720]: pgmap v288: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2023-07-14T18:24:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:09 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.805+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a500fcc00 con 0x7f8a3c0a11e0 2023-07-14T18:24:09.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.820+0000 7f8a477fe700 1 -- 172.21.15.64:0/2735983124 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+425238 (secure 0 0 0) 0x7f8a500fcc00 con 0x7f8a3c0a11e0 2023-07-14T18:24:09.876 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 msgr2=0x7f8a3c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 0x7f8a3c0a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8a501962a0 tx=0x7f8a38009260 comp rx=0 tx=0).stop 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 msgr2=0x7f8a50194cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000d950 tx=0x7f8a4000dc60 comp rx=0 tx=0).stop 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 shutdown_connections 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8a50103ee0 0x7f8a501985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a3c0a11e0 0x7f8a3c0a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a500fee40 0x7f8a50194cc0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 --2- 172.21.15.64:0/2735983124 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8a500ff7e0 0x7f8a50195200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 >> 172.21.15.64:0/2735983124 conn(0x7f8a50075ca0 msgr2=0x7f8a500fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 shutdown_connections 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:09.824+0000 7f8a54d4b700 1 -- 172.21.15.64:0/2735983124 wait complete. 2023-07-14T18:24:09.880 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:24:09.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:09 smithi144 ceph-mon[98417]: pgmap v288: 129 pgs: 129 active+clean; 641 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2023-07-14T18:24:09.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:09.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:09 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:10.324 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2023-07-14T18:24:09.474366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":566,"num_read_kb":405,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337628,"kb_used_data":13388,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604196,"statfs":{"total":1151940427776,"available":1151594696704,"internally_reserved":0,"allocated":13709312,"data_stored":7548379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":48,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002314"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405474+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:24:07.405474+0000","last_peered":"2023-07-14T18:24:07.405474+0000","last_clean":"2023-07-14T18:24:07.405474+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:24:07.405474+0000","last_undegraded":"2023-07-14T18:24:07.405474+0000","last_fullsized":"2023-07-14T18:24:07.405474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:24:11.348750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226781+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:23:11.226781+0000","last_peered":"2023-07-14T18:23:11.226781+0000","last_clean":"2023-07-14T18:23:11.226781+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:23:11.226781+0000","last_undegraded":"2023-07-14T18:23:11.226781+0000","last_fullsized":"2023-07-14T18:23:11.226781+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:58.662673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226533+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:23:11.226533+0000","last_peered":"2023-07-14T18:23:11.226533+0000","last_clean":"2023-07-14T18:23:11.226533+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:23:11.226533+0000","last_undegraded":"2023-07-14T18:23:11.226533+0000","last_fullsized":"2023-07-14T18:23:11.226533+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:50.079766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227979+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:23:11.227979+0000","last_peered":"2023-07-14T18:23:11.227979+0000","last_clean":"2023-07-14T18:23:11.227979+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:23:11.227979+0000","last_undegraded":"2023-07-14T18:23:11.227979+0000","last_fullsized":"2023-07-14T18:23:11.227979+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:42:06.130874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227740+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:23:11.227740+0000","last_peered":"2023-07-14T18:23:11.227740+0000","last_clean":"2023-07-14T18:23:11.227740+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:23:11.227740+0000","last_undegraded":"2023-07-14T18:23:11.227740+0000","last_fullsized":"2023-07-14T18:23:11.227740+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:12.302519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406070+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:24:06.406070+0000","last_peered":"2023-07-14T18:24:06.406070+0000","last_clean":"2023-07-14T18:24:06.406070+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:24:06.406070+0000","last_undegraded":"2023-07-14T18:24:06.406070+0000","last_fullsized":"2023-07-14T18:24:06.406070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:42.667371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405673+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:24:07.405673+0000","last_peered":"2023-07-14T18:24:07.405673+0000","last_clean":"2023-07-14T18:24:07.405673+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:24:07.405673+0000","last_undegraded":"2023-07-14T18:24:07.405673+0000","last_fullsized":"2023-07-14T18:24:07.405673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:14.526417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597894+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:24:07.597894+0000","last_peered":"2023-07-14T18:24:07.597894+0000","last_clean":"2023-07-14T18:24:07.597894+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:24:07.597894+0000","last_undegraded":"2023-07-14T18:24:07.597894+0000","last_fullsized":"2023-07-14T18:24:07.597894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:47.842854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.448033+0000","last_change":"2023-07-14T18:23:05.248889+0000","last_active":"2023-07-14T18:23:10.448033+0000","last_peered":"2023-07-14T18:23:10.448033+0000","last_clean":"2023-07-14T18:23:10.448033+0000","last_became_active":"2023-07-14T18:23:05.248500+0000","last_became_peered":"2023-07-14T18:23:05.248500+0000","last_unstale":"2023-07-14T18:23:10.448033+0000","last_undegraded":"2023-07-14T18:23:10.448033+0000","last_fullsized":"2023-07-14T18:23:10.448033+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:53:30.313031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.405714+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:24:06.405714+0000","last_peered":"2023-07-14T18:24:06.405714+0000","last_clean":"2023-07-14T18:24:06.405714+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:24:06.405714+0000","last_undegraded":"2023-07-14T18:24:06.405714+0000","last_fullsized":"2023-07-14T18:24:06.405714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:19:06.971461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406490+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:24:07.406490+0000","last_peered":"2023-07-14T18:24:07.406490+0000","last_clean":"2023-07-14T18:24:07.406490+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:24:07.406490+0000","last_undegraded":"2023-07-14T18:24:07.406490+0000","last_fullsized":"2023-07-14T18:24:07.406490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:44.364403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227886+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:23:11.227886+0000","last_peered":"2023-07-14T18:23:11.227886+0000","last_clean":"2023-07-14T18:23:11.227886+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:23:11.227886+0000","last_undegraded":"2023-07-14T18:23:11.227886+0000","last_fullsized":"2023-07-14T18:23:11.227886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:15.483937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820341+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:24:07.820341+0000","last_peered":"2023-07-14T18:24:07.820341+0000","last_clean":"2023-07-14T18:24:07.820341+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:24:07.820341+0000","last_undegraded":"2023-07-14T18:24:07.820341+0000","last_fullsized":"2023-07-14T18:24:07.820341+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:47.802767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226320+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:23:11.226320+0000","last_peered":"2023-07-14T18:23:11.226320+0000","last_clean":"2023-07-14T18:23:11.226320+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:23:11.226320+0000","last_undegraded":"2023-07-14T18:23:11.226320+0000","last_fullsized":"2023-07-14T18:23:11.226320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:40.019875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226467+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:23:11.226467+0000","last_peered":"2023-07-14T18:23:11.226467+0000","last_clean":"2023-07-14T18:23:11.226467+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:23:11.226467+0000","last_undegraded":"2023-07-14T18:23:11.226467+0000","last_fullsized":"2023-07-14T18:23:11.226467+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:46.851772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447060+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:23:10.447060+0000","last_peered":"2023-07-14T18:23:10.447060+0000","last_clean":"2023-07-14T18:23:10.447060+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:23:10.447060+0000","last_undegraded":"2023-07-14T18:23:10.447060+0000","last_fullsized":"2023-07-14T18:23:10.447060+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:57.018695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405475+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:24:07.405475+0000","last_peered":"2023-07-14T18:24:07.405475+0000","last_clean":"2023-07-14T18:24:07.405475+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:24:07.405475+0000","last_undegraded":"2023-07-14T18:24:07.405475+0000","last_fullsized":"2023-07-14T18:24:07.405475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:27:22.438148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407722+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:24:07.407722+0000","last_peered":"2023-07-14T18:24:07.407722+0000","last_clean":"2023-07-14T18:24:07.407722+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:24:07.407722+0000","last_undegraded":"2023-07-14T18:24:07.407722+0000","last_fullsized":"2023-07-14T18:24:07.407722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:18.810920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407624+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:24:07.407624+0000","last_peered":"2023-07-14T18:24:07.407624+0000","last_clean":"2023-07-14T18:24:07.407624+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:24:07.407624+0000","last_undegraded":"2023-07-14T18:24:07.407624+0000","last_fullsized":"2023-07-14T18:24:07.407624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:43.490417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406996+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:24:07.406996+0000","last_peered":"2023-07-14T18:24:07.406996+0000","last_clean":"2023-07-14T18:24:07.406996+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:24:07.406996+0000","last_undegraded":"2023-07-14T18:24:07.406996+0000","last_fullsized":"2023-07-14T18:24:07.406996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:27:08.177654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407692+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:24:07.407692+0000","last_peered":"2023-07-14T18:24:07.407692+0000","last_clean":"2023-07-14T18:24:07.407692+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:24:07.407692+0000","last_undegraded":"2023-07-14T18:24:07.407692+0000","last_fullsized":"2023-07-14T18:24:07.407692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:37.412855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598331+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:24:07.598331+0000","last_peered":"2023-07-14T18:24:07.598331+0000","last_clean":"2023-07-14T18:24:07.598331+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:24:07.598331+0000","last_undegraded":"2023-07-14T18:24:07.598331+0000","last_fullsized":"2023-07-14T18:24:07.598331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:51.649314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226877+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:23:11.226877+0000","last_peered":"2023-07-14T18:23:11.226877+0000","last_clean":"2023-07-14T18:23:11.226877+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:23:11.226877+0000","last_undegraded":"2023-07-14T18:23:11.226877+0000","last_fullsized":"2023-07-14T18:23:11.226877+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:54:59.578019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598686+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:24:07.598686+0000","last_peered":"2023-07-14T18:24:07.598686+0000","last_clean":"2023-07-14T18:24:07.598686+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:24:07.598686+0000","last_undegraded":"2023-07-14T18:24:07.598686+0000","last_fullsized":"2023-07-14T18:24:07.598686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.150320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226855+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:23:11.226855+0000","last_peered":"2023-07-14T18:23:11.226855+0000","last_clean":"2023-07-14T18:23:11.226855+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:23:11.226855+0000","last_undegraded":"2023-07-14T18:23:11.226855+0000","last_fullsized":"2023-07-14T18:23:11.226855+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:09:13.892292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227338+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:23:11.227338+0000","last_peered":"2023-07-14T18:23:11.227338+0000","last_clean":"2023-07-14T18:23:11.227338+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:23:11.227338+0000","last_undegraded":"2023-07-14T18:23:11.227338+0000","last_fullsized":"2023-07-14T18:23:11.227338+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:36:55.710853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.285894+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:23:28.285894+0000","last_peered":"2023-07-14T18:23:28.285894+0000","last_clean":"2023-07-14T18:23:28.285894+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:23:28.285894+0000","last_undegraded":"2023-07-14T18:23:28.285894+0000","last_fullsized":"2023-07-14T18:23:28.285894+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:30.855601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447648+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:23:10.447648+0000","last_peered":"2023-07-14T18:23:10.447648+0000","last_clean":"2023-07-14T18:23:10.447648+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:23:10.447648+0000","last_undegraded":"2023-07-14T18:23:10.447648+0000","last_fullsized":"2023-07-14T18:23:10.447648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:35.121798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227220+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:23:11.227220+0000","last_peered":"2023-07-14T18:23:11.227220+0000","last_clean":"2023-07-14T18:23:11.227220+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:23:11.227220+0000","last_undegraded":"2023-07-14T18:23:11.227220+0000","last_fullsized":"2023-07-14T18:23:11.227220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:10.747246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270798+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:23:27.270798+0000","last_peered":"2023-07-14T18:23:27.270798+0000","last_clean":"2023-07-14T18:23:27.270798+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:23:27.270798+0000","last_undegraded":"2023-07-14T18:23:27.270798+0000","last_fullsized":"2023-07-14T18:23:27.270798+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:38.727870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409769+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:24:07.409769+0000","last_peered":"2023-07-14T18:24:07.409769+0000","last_clean":"2023-07-14T18:24:07.409769+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:24:07.409769+0000","last_undegraded":"2023-07-14T18:24:07.409769+0000","last_fullsized":"2023-07-14T18:24:07.409769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:07.113007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"activating","last_fresh":"2023-07-14T18:24:07.408312+0000","last_change":"2023-07-14T18:24:07.407003+0000","last_active":"2023-07-14T18:24:06.405140+0000","last_peered":"2023-07-14T18:23:12.490237+0000","last_clean":"2023-07-14T18:23:12.490237+0000","last_became_active":"2023-07-14T18:23:09.448108+0000","last_became_peered":"2023-07-14T18:23:09.448108+0000","last_unstale":"2023-07-14T18:24:07.408312+0000","last_undegraded":"2023-07-14T18:24:07.408312+0000","last_fullsized":"2023-07-14T18:24:07.408312+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:34.685911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7,11],"acting":[2,10,7,11],"avail_no_missing":["2(0)","5(1)","7(2)","10(1)","11(3)"],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410924+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:24:07.410924+0000","last_peered":"2023-07-14T18:24:07.410924+0000","last_clean":"2023-07-14T18:24:07.410924+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:24:07.410924+0000","last_undegraded":"2023-07-14T18:24:07.410924+0000","last_fullsized":"2023-07-14T18:24:07.410924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:31.697182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226906+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:23:11.226906+0000","last_peered":"2023-07-14T18:23:11.226906+0000","last_clean":"2023-07-14T18:23:11.226906+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:23:11.226906+0000","last_undegraded":"2023-07-14T18:23:11.226906+0000","last_fullsized":"2023-07-14T18:23:11.226906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:51:56.191875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407774+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:24:07.407774+0000","last_peered":"2023-07-14T18:24:07.407774+0000","last_clean":"2023-07-14T18:24:07.407774+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:24:07.407774+0000","last_undegraded":"2023-07-14T18:24:07.407774+0000","last_fullsized":"2023-07-14T18:24:07.407774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:27.518543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.405996+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:24:06.405996+0000","last_peered":"2023-07-14T18:24:06.405996+0000","last_clean":"2023-07-14T18:24:06.405996+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:24:06.405996+0000","last_undegraded":"2023-07-14T18:24:06.405996+0000","last_fullsized":"2023-07-14T18:24:06.405996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:38:27.267958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.822326+0000","last_change":"2023-07-14T18:24:07.822326+0000","last_active":"2023-07-14T18:24:07.822326+0000","last_peered":"2023-07-14T18:24:07.822326+0000","last_clean":"2023-07-14T18:24:07.822326+0000","last_became_active":"2023-07-14T18:24:07.821961+0000","last_became_peered":"2023-07-14T18:24:07.821961+0000","last_unstale":"2023-07-14T18:24:07.822326+0000","last_undegraded":"2023-07-14T18:24:07.822326+0000","last_fullsized":"2023-07-14T18:24:07.822326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:49.138102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405293+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:24:07.405293+0000","last_peered":"2023-07-14T18:24:07.405293+0000","last_clean":"2023-07-14T18:24:07.405293+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:24:07.405293+0000","last_undegraded":"2023-07-14T18:24:07.405293+0000","last_fullsized":"2023-07-14T18:24:07.405293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:36.010004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820213+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:24:07.820213+0000","last_peered":"2023-07-14T18:24:07.820213+0000","last_clean":"2023-07-14T18:24:07.820213+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:24:07.820213+0000","last_undegraded":"2023-07-14T18:24:07.820213+0000","last_fullsized":"2023-07-14T18:24:07.820213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:32.445990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598031+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:24:07.598031+0000","last_peered":"2023-07-14T18:24:07.598031+0000","last_clean":"2023-07-14T18:24:07.598031+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:24:07.598031+0000","last_undegraded":"2023-07-14T18:24:07.598031+0000","last_fullsized":"2023-07-14T18:24:07.598031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:38.928448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226340+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:23:11.226340+0000","last_peered":"2023-07-14T18:23:11.226340+0000","last_clean":"2023-07-14T18:23:11.226340+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:23:11.226340+0000","last_undegraded":"2023-07-14T18:23:11.226340+0000","last_fullsized":"2023-07-14T18:23:11.226340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:39:06.464493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410048+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:24:07.410048+0000","last_peered":"2023-07-14T18:24:07.410048+0000","last_clean":"2023-07-14T18:24:07.410048+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:24:07.410048+0000","last_undegraded":"2023-07-14T18:24:07.410048+0000","last_fullsized":"2023-07-14T18:24:07.410048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:18.846973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407999+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:24:07.407999+0000","last_peered":"2023-07-14T18:24:07.407999+0000","last_clean":"2023-07-14T18:24:07.407999+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:24:07.407999+0000","last_undegraded":"2023-07-14T18:24:07.407999+0000","last_fullsized":"2023-07-14T18:24:07.407999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:36:19.820805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597965+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:24:07.597965+0000","last_peered":"2023-07-14T18:24:07.597965+0000","last_clean":"2023-07-14T18:24:07.597965+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:24:07.597965+0000","last_undegraded":"2023-07-14T18:24:07.597965+0000","last_fullsized":"2023-07-14T18:24:07.597965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:40.703098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":106,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406597+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:24:06.406597+0000","last_peered":"2023-07-14T18:24:06.406597+0000","last_clean":"2023-07-14T18:24:06.406597+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:24:06.406597+0000","last_undegraded":"2023-07-14T18:24:06.406597+0000","last_fullsized":"2023-07-14T18:24:06.406597+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:19.921934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447741+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:23:10.447741+0000","last_peered":"2023-07-14T18:23:10.447741+0000","last_clean":"2023-07-14T18:23:10.447741+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:23:10.447741+0000","last_undegraded":"2023-07-14T18:23:10.447741+0000","last_fullsized":"2023-07-14T18:23:10.447741+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:46.573175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.443969+0000","last_change":"2023-07-14T18:23:09.450239+0000","last_active":"2023-07-14T18:23:10.443969+0000","last_peered":"2023-07-14T18:23:10.443969+0000","last_clean":"2023-07-14T18:23:10.443969+0000","last_became_active":"2023-07-14T18:23:09.449827+0000","last_became_peered":"2023-07-14T18:23:09.449827+0000","last_unstale":"2023-07-14T18:23:10.443969+0000","last_undegraded":"2023-07-14T18:23:10.443969+0000","last_fullsized":"2023-07-14T18:23:10.443969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:45:48.081784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406710+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:24:07.406710+0000","last_peered":"2023-07-14T18:24:07.406710+0000","last_clean":"2023-07-14T18:24:07.406710+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:24:07.406710+0000","last_undegraded":"2023-07-14T18:24:07.406710+0000","last_fullsized":"2023-07-14T18:24:07.406710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:32:56.571997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226974+0000","last_change":"2023-07-14T18:23:05.255321+0000","last_active":"2023-07-14T18:23:11.226974+0000","last_peered":"2023-07-14T18:23:11.226974+0000","last_clean":"2023-07-14T18:23:11.226974+0000","last_became_active":"2023-07-14T18:23:05.254443+0000","last_became_peered":"2023-07-14T18:23:05.254443+0000","last_unstale":"2023-07-14T18:23:11.226974+0000","last_undegraded":"2023-07-14T18:23:11.226974+0000","last_fullsized":"2023-07-14T18:23:11.226974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:47:37.938549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227625+0000","last_change":"2023-07-14T18:23:09.448642+0000","last_active":"2023-07-14T18:23:11.227625+0000","last_peered":"2023-07-14T18:23:11.227625+0000","last_clean":"2023-07-14T18:23:11.227625+0000","last_became_active":"2023-07-14T18:23:09.448079+0000","last_became_peered":"2023-07-14T18:23:09.448079+0000","last_unstale":"2023-07-14T18:23:11.227625+0000","last_undegraded":"2023-07-14T18:23:11.227625+0000","last_fullsized":"2023-07-14T18:23:11.227625+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:25:39.875566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409637+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:24:07.409637+0000","last_peered":"2023-07-14T18:24:07.409637+0000","last_clean":"2023-07-14T18:24:07.409637+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:24:07.409637+0000","last_undegraded":"2023-07-14T18:24:07.409637+0000","last_fullsized":"2023-07-14T18:24:07.409637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:25.197615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447624+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:23:10.447624+0000","last_peered":"2023-07-14T18:23:10.447624+0000","last_clean":"2023-07-14T18:23:10.447624+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:23:10.447624+0000","last_undegraded":"2023-07-14T18:23:10.447624+0000","last_fullsized":"2023-07-14T18:23:10.447624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:01.328580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407780+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:24:07.407780+0000","last_peered":"2023-07-14T18:24:07.407780+0000","last_clean":"2023-07-14T18:24:07.407780+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:24:07.407780+0000","last_undegraded":"2023-07-14T18:24:07.407780+0000","last_fullsized":"2023-07-14T18:24:07.407780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:42.309537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'33","reported_seq":141,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406505+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:24:06.406505+0000","last_peered":"2023-07-14T18:24:06.406505+0000","last_clean":"2023-07-14T18:24:06.406505+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:24:06.406505+0000","last_undegraded":"2023-07-14T18:24:06.406505+0000","last_fullsized":"2023-07-14T18:24:06.406505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:10.210991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":98,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.228668+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:23:11.228668+0000","last_peered":"2023-07-14T18:23:11.228668+0000","last_clean":"2023-07-14T18:23:11.228668+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:23:11.228668+0000","last_undegraded":"2023-07-14T18:23:11.228668+0000","last_fullsized":"2023-07-14T18:23:11.228668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:23.893253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410030+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:24:07.410030+0000","last_peered":"2023-07-14T18:24:07.410030+0000","last_clean":"2023-07-14T18:24:07.410030+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:24:07.410030+0000","last_undegraded":"2023-07-14T18:24:07.410030+0000","last_fullsized":"2023-07-14T18:24:07.410030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:15.831940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407198+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:24:07.407198+0000","last_peered":"2023-07-14T18:24:07.407198+0000","last_clean":"2023-07-14T18:24:07.407198+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:24:07.407198+0000","last_undegraded":"2023-07-14T18:24:07.407198+0000","last_fullsized":"2023-07-14T18:24:07.407198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:03.898501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820320+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:24:07.820320+0000","last_peered":"2023-07-14T18:24:07.820320+0000","last_clean":"2023-07-14T18:24:07.820320+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:24:07.820320+0000","last_undegraded":"2023-07-14T18:24:07.820320+0000","last_fullsized":"2023-07-14T18:24:07.820320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:28:38.028092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405119+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:24:07.405119+0000","last_peered":"2023-07-14T18:24:07.405119+0000","last_clean":"2023-07-14T18:24:07.405119+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:24:07.405119+0000","last_undegraded":"2023-07-14T18:24:07.405119+0000","last_fullsized":"2023-07-14T18:24:07.405119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:57.627580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409613+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:24:07.409613+0000","last_peered":"2023-07-14T18:24:07.409613+0000","last_clean":"2023-07-14T18:24:07.409613+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:24:07.409613+0000","last_undegraded":"2023-07-14T18:24:07.409613+0000","last_fullsized":"2023-07-14T18:24:07.409613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:13.349187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405811+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:24:07.405811+0000","last_peered":"2023-07-14T18:24:07.405811+0000","last_clean":"2023-07-14T18:24:07.405811+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:24:07.405811+0000","last_undegraded":"2023-07-14T18:24:07.405811+0000","last_fullsized":"2023-07-14T18:24:07.405811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:17:48.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407367+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:24:07.407367+0000","last_peered":"2023-07-14T18:24:07.407367+0000","last_clean":"2023-07-14T18:24:07.407367+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:24:07.407367+0000","last_undegraded":"2023-07-14T18:24:07.407367+0000","last_fullsized":"2023-07-14T18:24:07.407367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:31:02.627246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:24:03.350262+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:24:03.350262+0000","last_peered":"2023-07-14T18:24:03.350262+0000","last_clean":"2023-07-14T18:24:03.350262+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:24:03.350262+0000","last_undegraded":"2023-07-14T18:24:03.350262+0000","last_fullsized":"2023-07-14T18:24:03.350262+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:54:03.971193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":40,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405854+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:24:07.405854+0000","last_peered":"2023-07-14T18:24:07.405854+0000","last_clean":"2023-07-14T18:24:07.405854+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:24:07.405854+0000","last_undegraded":"2023-07-14T18:24:07.405854+0000","last_fullsized":"2023-07-14T18:24:07.405854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.422448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820996+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:24:07.820996+0000","last_peered":"2023-07-14T18:24:07.820996+0000","last_clean":"2023-07-14T18:24:07.820996+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:24:07.820996+0000","last_undegraded":"2023-07-14T18:24:07.820996+0000","last_fullsized":"2023-07-14T18:24:07.820996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:00.885274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270941+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:23:27.270941+0000","last_peered":"2023-07-14T18:23:27.270941+0000","last_clean":"2023-07-14T18:23:27.270941+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:23:27.270941+0000","last_undegraded":"2023-07-14T18:23:27.270941+0000","last_fullsized":"2023-07-14T18:23:27.270941+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:36:45.841170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413544+0000","last_change":"2023-07-14T18:24:07.413544+0000","last_active":"2023-07-14T18:24:07.413544+0000","last_peered":"2023-07-14T18:24:07.413544+0000","last_clean":"2023-07-14T18:24:07.413544+0000","last_became_active":"2023-07-14T18:24:07.413217+0000","last_became_peered":"2023-07-14T18:24:07.413217+0000","last_unstale":"2023-07-14T18:24:07.413544+0000","last_undegraded":"2023-07-14T18:24:07.413544+0000","last_fullsized":"2023-07-14T18:24:07.413544+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:24.792524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410040+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:24:07.410040+0000","last_peered":"2023-07-14T18:24:07.410040+0000","last_clean":"2023-07-14T18:24:07.410040+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:24:07.410040+0000","last_undegraded":"2023-07-14T18:24:07.410040+0000","last_fullsized":"2023-07-14T18:24:07.410040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:18.284526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.284458+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:23:28.284458+0000","last_peered":"2023-07-14T18:23:28.284458+0000","last_clean":"2023-07-14T18:23:28.284458+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:23:28.284458+0000","last_undegraded":"2023-07-14T18:23:28.284458+0000","last_fullsized":"2023-07-14T18:23:28.284458+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:18:33.993734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405246+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:24:07.405246+0000","last_peered":"2023-07-14T18:24:07.405246+0000","last_clean":"2023-07-14T18:24:07.405246+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:24:07.405246+0000","last_undegraded":"2023-07-14T18:24:07.405246+0000","last_fullsized":"2023-07-14T18:24:07.405246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:41.007811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226809+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:23:11.226809+0000","last_peered":"2023-07-14T18:23:11.226809+0000","last_clean":"2023-07-14T18:23:11.226809+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:23:11.226809+0000","last_undegraded":"2023-07-14T18:23:11.226809+0000","last_fullsized":"2023-07-14T18:23:11.226809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:44.909597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413261+0000","last_change":"2023-07-14T18:24:07.413261+0000","last_active":"2023-07-14T18:24:07.413261+0000","last_peered":"2023-07-14T18:24:07.413261+0000","last_clean":"2023-07-14T18:24:07.413261+0000","last_became_active":"2023-07-14T18:24:07.412759+0000","last_became_peered":"2023-07-14T18:24:07.412759+0000","last_unstale":"2023-07-14T18:24:07.413261+0000","last_undegraded":"2023-07-14T18:24:07.413261+0000","last_fullsized":"2023-07-14T18:24:07.413261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:09.725334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406205+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:24:06.406205+0000","last_peered":"2023-07-14T18:24:06.406205+0000","last_clean":"2023-07-14T18:24:06.406205+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:24:06.406205+0000","last_undegraded":"2023-07-14T18:24:06.406205+0000","last_fullsized":"2023-07-14T18:24:06.406205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:00.302894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820487+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:24:07.820487+0000","last_peered":"2023-07-14T18:24:07.820487+0000","last_clean":"2023-07-14T18:24:07.820487+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:24:07.820487+0000","last_undegraded":"2023-07-14T18:24:07.820487+0000","last_fullsized":"2023-07-14T18:24:07.820487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:09:47.913784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405859+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:24:07.405859+0000","last_peered":"2023-07-14T18:24:07.405859+0000","last_clean":"2023-07-14T18:24:07.405859+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:24:07.405859+0000","last_undegraded":"2023-07-14T18:24:07.405859+0000","last_fullsized":"2023-07-14T18:24:07.405859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:23.957543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406294+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:24:06.406294+0000","last_peered":"2023-07-14T18:24:06.406294+0000","last_clean":"2023-07-14T18:24:06.406294+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:24:06.406294+0000","last_undegraded":"2023-07-14T18:24:06.406294+0000","last_fullsized":"2023-07-14T18:24:06.406294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:32.953610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821088+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:24:07.821088+0000","last_peered":"2023-07-14T18:24:07.821088+0000","last_clean":"2023-07-14T18:24:07.821088+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:24:07.821088+0000","last_undegraded":"2023-07-14T18:24:07.821088+0000","last_fullsized":"2023-07-14T18:24:07.821088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:26.669137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410319+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:24:07.410319+0000","last_peered":"2023-07-14T18:24:07.410319+0000","last_clean":"2023-07-14T18:24:07.410319+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:24:07.410319+0000","last_undegraded":"2023-07-14T18:24:07.410319+0000","last_fullsized":"2023-07-14T18:24:07.410319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:11.487403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:08.355867+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:24:08.355867+0000","last_peered":"2023-07-14T18:24:08.355867+0000","last_clean":"2023-07-14T18:24:08.355867+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:24:08.355867+0000","last_undegraded":"2023-07-14T18:24:08.355867+0000","last_fullsized":"2023-07-14T18:24:08.355867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:44.654417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406370+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:24:07.406370+0000","last_peered":"2023-07-14T18:24:07.406370+0000","last_clean":"2023-07-14T18:24:07.406370+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:24:07.406370+0000","last_undegraded":"2023-07-14T18:24:07.406370+0000","last_fullsized":"2023-07-14T18:24:07.406370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:01.561467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:01.628715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:28.286159+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:23:28.286159+0000","last_peered":"2023-07-14T18:23:28.286159+0000","last_clean":"2023-07-14T18:23:28.286159+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:23:28.286159+0000","last_undegraded":"2023-07-14T18:23:28.286159+0000","last_fullsized":"2023-07-14T18:23:28.286159+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:58:52.919560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405706+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:24:07.405706+0000","last_peered":"2023-07-14T18:24:07.405706+0000","last_clean":"2023-07-14T18:24:07.405706+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:24:07.405706+0000","last_undegraded":"2023-07-14T18:24:07.405706+0000","last_fullsized":"2023-07-14T18:24:07.405706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:07.000585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410311+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:24:07.410311+0000","last_peered":"2023-07-14T18:24:07.410311+0000","last_clean":"2023-07-14T18:24:07.410311+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:24:07.410311+0000","last_undegraded":"2023-07-14T18:24:07.410311+0000","last_fullsized":"2023-07-14T18:24:07.410311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:22.806069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820058+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:24:07.820058+0000","last_peered":"2023-07-14T18:24:07.820058+0000","last_clean":"2023-07-14T18:24:07.820058+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:24:07.820058+0000","last_undegraded":"2023-07-14T18:24:07.820058+0000","last_fullsized":"2023-07-14T18:24:07.820058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:00.275183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405700+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:24:07.405700+0000","last_peered":"2023-07-14T18:24:07.405700+0000","last_clean":"2023-07-14T18:24:07.405700+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:24:07.405700+0000","last_undegraded":"2023-07-14T18:24:07.405700+0000","last_fullsized":"2023-07-14T18:24:07.405700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:04.719205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405561+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:24:07.405561+0000","last_peered":"2023-07-14T18:24:07.405561+0000","last_clean":"2023-07-14T18:24:07.405561+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:24:07.405561+0000","last_undegraded":"2023-07-14T18:24:07.405561+0000","last_fullsized":"2023-07-14T18:24:07.405561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:44.840479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270466+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:23:27.270466+0000","last_peered":"2023-07-14T18:23:27.270466+0000","last_clean":"2023-07-14T18:23:27.270466+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:23:27.270466+0000","last_undegraded":"2023-07-14T18:23:27.270466+0000","last_fullsized":"2023-07-14T18:23:27.270466+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:07:29.140961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820701+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:24:07.820701+0000","last_peered":"2023-07-14T18:24:07.820701+0000","last_clean":"2023-07-14T18:24:07.820701+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:24:07.820701+0000","last_undegraded":"2023-07-14T18:24:07.820701+0000","last_fullsized":"2023-07-14T18:24:07.820701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:52:09.481241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597956+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:24:07.597956+0000","last_peered":"2023-07-14T18:24:07.597956+0000","last_clean":"2023-07-14T18:24:07.597956+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:24:07.597956+0000","last_undegraded":"2023-07-14T18:24:07.597956+0000","last_fullsized":"2023-07-14T18:24:07.597956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:55.573942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407882+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:24:07.407882+0000","last_peered":"2023-07-14T18:24:07.407882+0000","last_clean":"2023-07-14T18:24:07.407882+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:24:07.407882+0000","last_undegraded":"2023-07-14T18:24:07.407882+0000","last_fullsized":"2023-07-14T18:24:07.407882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:55:27.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598511+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:24:07.598511+0000","last_peered":"2023-07-14T18:24:07.598511+0000","last_clean":"2023-07-14T18:24:07.598511+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:24:07.598511+0000","last_undegraded":"2023-07-14T18:24:07.598511+0000","last_fullsized":"2023-07-14T18:24:07.598511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:57.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406745+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:24:07.406745+0000","last_peered":"2023-07-14T18:24:07.406745+0000","last_clean":"2023-07-14T18:24:07.406745+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:24:07.406745+0000","last_undegraded":"2023-07-14T18:24:07.406745+0000","last_fullsized":"2023-07-14T18:24:07.406745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:40.891887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:27.270819+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:23:27.270819+0000","last_peered":"2023-07-14T18:23:27.270819+0000","last_clean":"2023-07-14T18:23:27.270819+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:23:27.270819+0000","last_undegraded":"2023-07-14T18:23:27.270819+0000","last_fullsized":"2023-07-14T18:23:27.270819+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:54:38.614796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226909+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:23:11.226909+0000","last_peered":"2023-07-14T18:23:11.226909+0000","last_clean":"2023-07-14T18:23:11.226909+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:23:11.226909+0000","last_undegraded":"2023-07-14T18:23:11.226909+0000","last_fullsized":"2023-07-14T18:23:11.226909+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:07:43.634763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227411+0000","last_change":"2023-07-14T18:23:09.449832+0000","last_active":"2023-07-14T18:23:11.227411+0000","last_peered":"2023-07-14T18:23:11.227411+0000","last_clean":"2023-07-14T18:23:11.227411+0000","last_became_active":"2023-07-14T18:23:09.448997+0000","last_became_peered":"2023-07-14T18:23:09.448997+0000","last_unstale":"2023-07-14T18:23:11.227411+0000","last_undegraded":"2023-07-14T18:23:11.227411+0000","last_fullsized":"2023-07-14T18:23:11.227411+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:02:56.933699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:24.242161+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:23:24.242161+0000","last_peered":"2023-07-14T18:23:24.242161+0000","last_clean":"2023-07-14T18:23:24.242161+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:23:24.242161+0000","last_undegraded":"2023-07-14T18:23:24.242161+0000","last_fullsized":"2023-07-14T18:23:24.242161+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:18:46.321986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226403+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:23:11.226403+0000","last_peered":"2023-07-14T18:23:11.226403+0000","last_clean":"2023-07-14T18:23:11.226403+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:23:11.226403+0000","last_undegraded":"2023-07-14T18:23:11.226403+0000","last_fullsized":"2023-07-14T18:23:11.226403+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:03:27.653641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407589+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:24:07.407589+0000","last_peered":"2023-07-14T18:24:07.407589+0000","last_clean":"2023-07-14T18:24:07.407589+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:24:07.407589+0000","last_undegraded":"2023-07-14T18:24:07.407589+0000","last_fullsized":"2023-07-14T18:24:07.407589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:02.479091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:37.977157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404952+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:24:07.404952+0000","last_peered":"2023-07-14T18:24:07.404952+0000","last_clean":"2023-07-14T18:24:07.404952+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:24:07.404952+0000","last_undegraded":"2023-07-14T18:24:07.404952+0000","last_fullsized":"2023-07-14T18:24:07.404952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:37.080797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227722+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:23:11.227722+0000","last_peered":"2023-07-14T18:23:11.227722+0000","last_clean":"2023-07-14T18:23:11.227722+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:23:11.227722+0000","last_undegraded":"2023-07-14T18:23:11.227722+0000","last_fullsized":"2023-07-14T18:23:11.227722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:26:45.518354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407086+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:24:07.407086+0000","last_peered":"2023-07-14T18:24:07.407086+0000","last_clean":"2023-07-14T18:24:07.407086+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:24:07.407086+0000","last_undegraded":"2023-07-14T18:24:07.407086+0000","last_fullsized":"2023-07-14T18:24:07.407086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:39.830952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598357+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:24:07.598357+0000","last_peered":"2023-07-14T18:24:07.598357+0000","last_clean":"2023-07-14T18:24:07.598357+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:24:07.598357+0000","last_undegraded":"2023-07-14T18:24:07.598357+0000","last_fullsized":"2023-07-14T18:24:07.598357+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:12.153021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.227371+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:23:11.227371+0000","last_peered":"2023-07-14T18:23:11.227371+0000","last_clean":"2023-07-14T18:23:11.227371+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:23:11.227371+0000","last_undegraded":"2023-07-14T18:23:11.227371+0000","last_fullsized":"2023-07-14T18:23:11.227371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:20.402932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.416915+0000","last_change":"2023-07-14T18:24:07.416556+0000","last_active":"2023-07-14T18:24:07.416915+0000","last_peered":"2023-07-14T18:24:07.416915+0000","last_clean":"2023-07-14T18:24:07.416915+0000","last_became_active":"2023-07-14T18:24:07.408596+0000","last_became_peered":"2023-07-14T18:24:07.408596+0000","last_unstale":"2023-07-14T18:24:07.416915+0000","last_undegraded":"2023-07-14T18:24:07.416915+0000","last_fullsized":"2023-07-14T18:24:07.416915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:10.303257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405892+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:24:07.405892+0000","last_peered":"2023-07-14T18:24:07.405892+0000","last_clean":"2023-07-14T18:24:07.405892+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:24:07.405892+0000","last_undegraded":"2023-07-14T18:24:07.405892+0000","last_fullsized":"2023-07-14T18:24:07.405892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:59:09.124887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447265+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:23:10.447265+0000","last_peered":"2023-07-14T18:23:10.447265+0000","last_clean":"2023-07-14T18:23:10.447265+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:23:10.447265+0000","last_undegraded":"2023-07-14T18:23:10.447265+0000","last_fullsized":"2023-07-14T18:23:10.447265+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:44:43.737345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407530+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:24:07.407530+0000","last_peered":"2023-07-14T18:24:07.407530+0000","last_clean":"2023-07-14T18:24:07.407530+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:24:07.407530+0000","last_undegraded":"2023-07-14T18:24:07.407530+0000","last_fullsized":"2023-07-14T18:24:07.407530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:33.188924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406610+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:24:07.406610+0000","last_peered":"2023-07-14T18:24:07.406610+0000","last_clean":"2023-07-14T18:24:07.406610+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:24:07.406610+0000","last_undegraded":"2023-07-14T18:24:07.406610+0000","last_fullsized":"2023-07-14T18:24:07.406610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:25:28.457835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226097+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:23:11.226097+0000","last_peered":"2023-07-14T18:23:11.226097+0000","last_clean":"2023-07-14T18:23:11.226097+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:23:11.226097+0000","last_undegraded":"2023-07-14T18:23:11.226097+0000","last_fullsized":"2023-07-14T18:23:11.226097+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:46:26.374670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409875+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:24:07.409875+0000","last_peered":"2023-07-14T18:24:07.409875+0000","last_clean":"2023-07-14T18:24:07.409875+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:24:07.409875+0000","last_undegraded":"2023-07-14T18:24:07.409875+0000","last_fullsized":"2023-07-14T18:24:07.409875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:21:05.734566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406230+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:24:07.406230+0000","last_peered":"2023-07-14T18:24:07.406230+0000","last_clean":"2023-07-14T18:24:07.406230+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:24:07.406230+0000","last_undegraded":"2023-07-14T18:24:07.406230+0000","last_fullsized":"2023-07-14T18:24:07.406230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:18.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408921+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:24:07.408921+0000","last_peered":"2023-07-14T18:24:07.408921+0000","last_clean":"2023-07-14T18:24:07.408921+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:24:07.408921+0000","last_undegraded":"2023-07-14T18:24:07.408921+0000","last_fullsized":"2023-07-14T18:24:07.408921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:45.492125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821139+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:24:07.821139+0000","last_peered":"2023-07-14T18:24:07.821139+0000","last_clean":"2023-07-14T18:24:07.821139+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:24:07.821139+0000","last_undegraded":"2023-07-14T18:24:07.821139+0000","last_fullsized":"2023-07-14T18:24:07.821139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:41.844783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407181+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:24:07.407181+0000","last_peered":"2023-07-14T18:24:07.407181+0000","last_clean":"2023-07-14T18:24:07.407181+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:24:07.407181+0000","last_undegraded":"2023-07-14T18:24:07.407181+0000","last_fullsized":"2023-07-14T18:24:07.407181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:34:35.248238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405025+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:24:07.405025+0000","last_peered":"2023-07-14T18:24:07.405025+0000","last_clean":"2023-07-14T18:24:07.405025+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:24:07.405025+0000","last_undegraded":"2023-07-14T18:24:07.405025+0000","last_fullsized":"2023-07-14T18:24:07.405025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:53.390475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406959+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:24:07.406959+0000","last_peered":"2023-07-14T18:24:07.406959+0000","last_clean":"2023-07-14T18:24:07.406959+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:24:07.406959+0000","last_undegraded":"2023-07-14T18:24:07.406959+0000","last_fullsized":"2023-07-14T18:24:07.406959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:26.096564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406915+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:24:07.406915+0000","last_peered":"2023-07-14T18:24:07.406915+0000","last_clean":"2023-07-14T18:24:07.406915+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:24:07.406915+0000","last_undegraded":"2023-07-14T18:24:07.406915+0000","last_fullsized":"2023-07-14T18:24:07.406915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:32:24.873296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:11.226339+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:23:11.226339+0000","last_peered":"2023-07-14T18:23:11.226339+0000","last_clean":"2023-07-14T18:23:11.226339+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:23:11.226339+0000","last_undegraded":"2023-07-14T18:23:11.226339+0000","last_fullsized":"2023-07-14T18:23:11.226339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:45.583600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820615+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:24:07.820615+0000","last_peered":"2023-07-14T18:24:07.820615+0000","last_clean":"2023-07-14T18:24:07.820615+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:24:07.820615+0000","last_undegraded":"2023-07-14T18:24:07.820615+0000","last_fullsized":"2023-07-14T18:24:07.820615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:23:26.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405289+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:24:07.405289+0000","last_peered":"2023-07-14T18:24:07.405289+0000","last_clean":"2023-07-14T18:24:07.405289+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:24:07.405289+0000","last_undegraded":"2023-07-14T18:24:07.405289+0000","last_fullsized":"2023-07-14T18:24:07.405289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:02.717582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407277+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:24:07.407277+0000","last_peered":"2023-07-14T18:24:07.407277+0000","last_clean":"2023-07-14T18:24:07.407277+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:24:07.407277+0000","last_undegraded":"2023-07-14T18:24:07.407277+0000","last_fullsized":"2023-07-14T18:24:07.407277+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:15:50.630547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598631+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:24:07.598631+0000","last_peered":"2023-07-14T18:24:07.598631+0000","last_clean":"2023-07-14T18:24:07.598631+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:24:07.598631+0000","last_undegraded":"2023-07-14T18:24:07.598631+0000","last_fullsized":"2023-07-14T18:24:07.598631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:06.982685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410189+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:24:07.410189+0000","last_peered":"2023-07-14T18:24:07.410189+0000","last_clean":"2023-07-14T18:24:07.410189+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:24:07.410189+0000","last_undegraded":"2023-07-14T18:24:07.410189+0000","last_fullsized":"2023-07-14T18:24:07.410189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:49:31.338811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598143+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:24:07.598143+0000","last_peered":"2023-07-14T18:24:07.598143+0000","last_clean":"2023-07-14T18:24:07.598143+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:24:07.598143+0000","last_undegraded":"2023-07-14T18:24:07.598143+0000","last_fullsized":"2023-07-14T18:24:07.598143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":18,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T18:23:10.447443+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:23:10.447443+0000","last_peered":"2023-07-14T18:23:10.447443+0000","last_clean":"2023-07-14T18:23:10.447443+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:23:10.447443+0000","last_undegraded":"2023-07-14T18:23:10.447443+0000","last_fullsized":"2023-07-14T18:23:10.447443+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:22.891172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820178+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:24:07.820178+0000","last_peered":"2023-07-14T18:24:07.820178+0000","last_clean":"2023-07-14T18:24:07.820178+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:24:07.820178+0000","last_undegraded":"2023-07-14T18:24:07.820178+0000","last_fullsized":"2023-07-14T18:24:07.820178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:24.150124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405223+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:24:07.405223+0000","last_peered":"2023-07-14T18:24:07.405223+0000","last_clean":"2023-07-14T18:24:07.405223+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:24:07.405223+0000","last_undegraded":"2023-07-14T18:24:07.405223+0000","last_fullsized":"2023-07-14T18:24:07.405223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:22.497297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":430,"num_read_kb":286,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514520,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":477638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.64400000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.66200000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.625},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68400000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72899999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.74099999999999999}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1103531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63,"15min":0.63},"min":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.78600000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.497}]},{"osd":4,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.433},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.58699999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.82999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.872,"5min":1.052,"15min":1.052},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.58099999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.68300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.92600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.80200000000000005},"min":{"1min":0.64600000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78800000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.57599999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.87,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84699999999999998,"5min":1.256,"15min":1.256},"last":0.72499999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.75800000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":471748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.499,"15min":0.46400000000000002},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":5.8559999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.5140000000000002,"5min":5.5140000000000002,"15min":5.5140000000000002},"last":0.624}]},{"osd":2,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.71299999999999997,"15min":0.68799999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.641,"5min":5.641,"15min":5.641},"last":0.35099999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.318}]},{"osd":4,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5679999999999996,"5min":5.5679999999999996,"15min":5.5679999999999996},"last":0.745},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.59699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78200000000000003,"15min":0.79400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.41999999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80400000000000005,"15min":0.81399999999999995},"min":{"1min":0.53000000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.68799999999999994}]},{"osd":8,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.85299999999999998,"15min":0.82899999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.875},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.53100000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.57499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":458520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42999999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.52400000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:22:21 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.79900000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.75}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.77500000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.73499999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70399999999999996,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.55200000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.91700000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.872,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.95399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78800000000000003}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1073683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.371},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.46400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.86899999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.38200000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89600000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.753},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.86199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92500000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90500000000000003,"5min":1.02,"15min":1.02},"last":0.624}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.77300000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97099999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.66600000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84499999999999997}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72599999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":472409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40500000000000003},"min":{"1min":0.28299999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60299999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93300000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.45800000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96399999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.63},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.80100000000000005,"5min":1.099,"15min":1.099},"last":0.501}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.56599999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.621}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76000000000000001,"15min":0.76500000000000001},"min":{"1min":0.63,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":513179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.70899999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.53400000000000003},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.749}]},{"osd":4,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.42999999999999999},"min":{"1min":0.35799999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71199999999999997,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.48499999999999999,"15min":0.46200000000000002},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.76700000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.68600000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.79400000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.64000000000000001,"15min":0.61299999999999999},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93999999999999995,"5min":1.089,"15min":1.089},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.64000000000000001,"15min":0.61499999999999999},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.498,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0249999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.623,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88900000000000001,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.82099999999999995}]},{"osd":8,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69499999999999995,"15min":0.67900000000000005},"min":{"1min":0.54600000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.012,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.80900000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":1.173,"15min":1.173},"last":0.93999999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.097,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.89900000000000002}]},{"osd":11,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.878},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.95999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":455171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56899999999999995,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.61199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81100000000000005,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.96599999999999997,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.629},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.432}]},{"osd":4,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.54700000000000004},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.503},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.61499999999999999,"15min":0.57099999999999995},"min":{"1min":0.498,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70599999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.48599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89800000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.45100000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:22:01 2023","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74299999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97399999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.68500000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.78500000000000003},"min":{"1min":0.503,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.85499999999999998,"5min":1.036,"15min":1.036},"last":0.80500000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.442,"15min":0.442},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.65500000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91400000000000003,"5min":1.075,"15min":1.075},"last":0.82299999999999995}]},{"osd":11,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.72099999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":523641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.374},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.54100000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.497},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.434}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.45200000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.72299999999999998,"15min":0.70499999999999996},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.58399999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.621,"15min":1.621},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.627}]},{"osd":11,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.54700000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":472157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41199999999999998,"15min":0.374},"min":{"1min":0.29099999999999998,"5min":0.161,"15min":0.161},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.47799999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.629,"15min":0.57899999999999996},"min":{"1min":0.41799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":8.1820000000000004},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.0289999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.68200000000000005,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.2210000000000001,"5min":8.2210000000000001,"15min":8.2210000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.93000000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":1.0049999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.68500000000000005,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.69199999999999995,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.81000000000000005,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":11.609999999999999,"5min":11.609999999999999,"15min":11.609999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.125,"5min":0.84499999999999997,"15min":0.79900000000000004},"min":{"1min":0.497,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.0559999999999992,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":1.044}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.28,"5min":0.879,"15min":0.81200000000000006},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.83699999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.90400000000000003,"15min":0.83799999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":12.291,"5min":12.291,"15min":12.291},"last":1.0629999999999999}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":11.263999999999999,"5min":11.263999999999999,"15min":11.263999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.1060000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.38100000000000001,"15min":0.36599999999999999},"min":{"1min":0.27500000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.59599999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.41099999999999998,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.38100000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.375,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68100000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.54600000000000004,"15min":0.52500000000000002},"min":{"1min":0.437,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.77200000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.55300000000000005,"15min":0.53000000000000003},"min":{"1min":0.40899999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.50600000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.67400000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89000000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.56899999999999995}]},{"osd":7,"last update":"Fri Jul 14 18:23:05 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60099999999999998,"15min":0.60799999999999998},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.89300000000000002,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58699999999999997}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.71099999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.56000000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.755,"15min":0.755},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53000000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1601536,"data_stored":1069303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.307},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.32500000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.45400000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.434},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83399999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52100000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68000000000000005,"15min":0.66000000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94499999999999995,"5min":1.075,"15min":1.075},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84699999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.47999999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.498}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89600000000000002,"5min":1.087,"15min":1.087},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.80200000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.98799999999999999,"5min":1.206,"15min":1.206},"last":0.57699999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95699999999999996,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77100000000000002},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.873,"5min":1.175,"15min":1.175},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:24:10.333 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:24:10.901 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:11.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:11.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:11.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:24:11.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:24:11.062 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:10 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.092 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:11.092 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:11.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:24:11.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:24:11.093 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:10 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: pgmap v289: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:11.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:11.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:11.184 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:24:11.184 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:24:11.184 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:10 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:24:12.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 -- 172.21.15.64:0/3460793412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac101830 msgr2=0x7f64ac109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:12.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 --2- 172.21.15.64:0/3460793412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac101830 0x7f64ac109d30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009990 tx=0x7f64a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 -- 172.21.15.64:0/3460793412 shutdown_connections 2023-07-14T18:24:12.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 --2- 172.21.15.64:0/3460793412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.223 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 --2- 172.21.15.64:0/3460793412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac10a270 0x7f64ac10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 --2- 172.21.15.64:0/3460793412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac101830 0x7f64ac109d30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 -- 172.21.15.64:0/3460793412 >> 172.21.15.64:0/3460793412 conn(0x7f64ac075ca0 msgr2=0x7f64ac0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:12.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 -- 172.21.15.64:0/3460793412 shutdown_connections 2023-07-14T18:24:12.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 -- 172.21.15.64:0/3460793412 wait complete. 2023-07-14T18:24:12.224 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.169+0000 7f64b183e700 1 Processor -- start 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 -- start start 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac101830 0x7f64ac198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19cdc0 con 0x7f64ac10a270 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19cf30 con 0x7f64ac101830 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64b183e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64ac19d0f0 con 0x7f64ac100f20 2023-07-14T18:24:12.225 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47908/0 (socket says 172.21.15.64:47908) 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 -- 172.21.15.64:0/1746011224 learned_addr learned my addr 172.21.15.64:0/1746011224 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64aa7fc700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac101830 0x7f64ac198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64aaffd700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 -- 172.21.15.64:0/1746011224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 msgr2=0x7f64ac197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 -- 172.21.15.64:0/1746011224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac101830 msgr2=0x7f64ac198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:12.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac101830 0x7f64ac198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64ab7fe700 1 -- 172.21.15.64:0/1746011224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a00095f0 con 0x7f64ac10a270 2023-07-14T18:24:12.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.170+0000 7f64aaffd700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:12.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64ab7fe700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ab50 tx=0x7f64a400ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:12.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a400f070 con 0x7f64ac10a270 2023-07-14T18:24:12.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f64a4007a80 con 0x7f64ac10a270 2023-07-14T18:24:12.228 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64ac19d3d0 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4005280 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.171+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64ac1a4c30 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.172+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64ac1077f0 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.173+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f64a4004ad0 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.174+0000 7f64b083c700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 0x7f64940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.174+0000 7f64aaffd700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 0x7f64940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.174+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f64a40cc890 con 0x7f64ac10a270 2023-07-14T18:24:12.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.174+0000 7f64aaffd700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 0x7f64940a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f649c005fd0 tx=0x7f649c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:12.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.177+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f64a4095ba0 con 0x7f64ac10a270 2023-07-14T18:24:12.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.369+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64ac060a80 con 0x7f64940a11e0 2023-07-14T18:24:12.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.383+0000 7f64b083c700 1 -- 172.21.15.64:0/1746011224 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426046 (secure 0 0 0) 0x7f64ac060a80 con 0x7f64940a11e0 2023-07-14T18:24:12.481 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:12.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 msgr2=0x7f64940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:12.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 0x7f64940a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f649c005fd0 tx=0x7f649c005e30 comp rx=0 tx=0).stop 2023-07-14T18:24:12.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 msgr2=0x7f64ac19c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f64a400ab50 tx=0x7f64a400ae60 comp rx=0 tx=0).stop 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 shutdown_connections 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f64ac100f20 0x7f64ac197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f64940a11e0 0x7f64940a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f64ac10a270 0x7f64ac19c7b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 --2- 172.21.15.64:0/1746011224 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f64ac101830 0x7f64ac198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:12.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 >> 172.21.15.64:0/1746011224 conn(0x7f64ac075ca0 msgr2=0x7f64ac0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:12.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 shutdown_connections 2023-07-14T18:24:12.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:12.387+0000 7f64b183e700 1 -- 172.21.15.64:0/1746011224 wait complete. 2023-07-14T18:24:12.486 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:24:12.852 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2023-07-14T18:24:11.475041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":571,"num_read_kb":405,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337772,"kb_used_data":13516,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604052,"statfs":{"total":1151940427776,"available":1151594549248,"internally_reserved":0,"allocated":13840384,"data_stored":7624715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":63,"num_read_kb":28,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002400"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405474+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:24:07.405474+0000","last_peered":"2023-07-14T18:24:07.405474+0000","last_clean":"2023-07-14T18:24:07.405474+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:24:07.405474+0000","last_undegraded":"2023-07-14T18:24:07.405474+0000","last_fullsized":"2023-07-14T18:24:07.405474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:24:11.348750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405925+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:24:07.405925+0000","last_peered":"2023-07-14T18:24:07.405925+0000","last_clean":"2023-07-14T18:24:07.405925+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:24:07.405925+0000","last_undegraded":"2023-07-14T18:24:07.405925+0000","last_fullsized":"2023-07-14T18:24:07.405925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:08.123484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405720+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:24:07.405720+0000","last_peered":"2023-07-14T18:24:07.405720+0000","last_clean":"2023-07-14T18:24:07.405720+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:24:07.405720+0000","last_undegraded":"2023-07-14T18:24:07.405720+0000","last_fullsized":"2023-07-14T18:24:07.405720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:18.329769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410214+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:24:07.410214+0000","last_peered":"2023-07-14T18:24:07.410214+0000","last_clean":"2023-07-14T18:24:07.410214+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:24:07.410214+0000","last_undegraded":"2023-07-14T18:24:07.410214+0000","last_fullsized":"2023-07-14T18:24:07.410214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:35.873720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403632+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:24:07.403632+0000","last_peered":"2023-07-14T18:24:07.403632+0000","last_clean":"2023-07-14T18:24:07.403632+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:24:07.403632+0000","last_undegraded":"2023-07-14T18:24:07.403632+0000","last_fullsized":"2023-07-14T18:24:07.403632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:47.188696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406070+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:24:06.406070+0000","last_peered":"2023-07-14T18:24:06.406070+0000","last_clean":"2023-07-14T18:24:06.406070+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:24:06.406070+0000","last_undegraded":"2023-07-14T18:24:06.406070+0000","last_fullsized":"2023-07-14T18:24:06.406070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:54:42.667371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405673+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:24:07.405673+0000","last_peered":"2023-07-14T18:24:07.405673+0000","last_clean":"2023-07-14T18:24:07.405673+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:24:07.405673+0000","last_undegraded":"2023-07-14T18:24:07.405673+0000","last_fullsized":"2023-07-14T18:24:07.405673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:14.526417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597894+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:24:07.597894+0000","last_peered":"2023-07-14T18:24:07.597894+0000","last_clean":"2023-07-14T18:24:07.597894+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:24:07.597894+0000","last_undegraded":"2023-07-14T18:24:07.597894+0000","last_fullsized":"2023-07-14T18:24:07.597894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:47.842854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413213+0000","last_change":"2023-07-14T18:24:07.413213+0000","last_active":"2023-07-14T18:24:07.413213+0000","last_peered":"2023-07-14T18:24:07.413213+0000","last_clean":"2023-07-14T18:24:07.413213+0000","last_became_active":"2023-07-14T18:24:07.412865+0000","last_became_peered":"2023-07-14T18:24:07.412865+0000","last_unstale":"2023-07-14T18:24:07.413213+0000","last_undegraded":"2023-07-14T18:24:07.413213+0000","last_fullsized":"2023-07-14T18:24:07.413213+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:33.198078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.405714+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:24:06.405714+0000","last_peered":"2023-07-14T18:24:06.405714+0000","last_clean":"2023-07-14T18:24:06.405714+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:24:06.405714+0000","last_undegraded":"2023-07-14T18:24:06.405714+0000","last_fullsized":"2023-07-14T18:24:06.405714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:19:06.971461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406490+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:24:07.406490+0000","last_peered":"2023-07-14T18:24:07.406490+0000","last_clean":"2023-07-14T18:24:07.406490+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:24:07.406490+0000","last_undegraded":"2023-07-14T18:24:07.406490+0000","last_fullsized":"2023-07-14T18:24:07.406490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:44.364403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410203+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:24:07.410203+0000","last_peered":"2023-07-14T18:24:07.410203+0000","last_clean":"2023-07-14T18:24:07.410203+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:24:07.410203+0000","last_undegraded":"2023-07-14T18:24:07.410203+0000","last_fullsized":"2023-07-14T18:24:07.410203+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:33.517728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820341+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:24:07.820341+0000","last_peered":"2023-07-14T18:24:07.820341+0000","last_clean":"2023-07-14T18:24:07.820341+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:24:07.820341+0000","last_undegraded":"2023-07-14T18:24:07.820341+0000","last_fullsized":"2023-07-14T18:24:07.820341+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:47.802767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405516+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:24:07.405516+0000","last_peered":"2023-07-14T18:24:07.405516+0000","last_clean":"2023-07-14T18:24:07.405516+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:24:07.405516+0000","last_undegraded":"2023-07-14T18:24:07.405516+0000","last_fullsized":"2023-07-14T18:24:07.405516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:48:51.690671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405517+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:24:07.405517+0000","last_peered":"2023-07-14T18:24:07.405517+0000","last_clean":"2023-07-14T18:24:07.405517+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:24:07.405517+0000","last_undegraded":"2023-07-14T18:24:07.405517+0000","last_fullsized":"2023-07-14T18:24:07.405517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:59.951987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407820+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:24:07.407820+0000","last_peered":"2023-07-14T18:24:07.407820+0000","last_clean":"2023-07-14T18:24:07.407820+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:24:07.407820+0000","last_undegraded":"2023-07-14T18:24:07.407820+0000","last_fullsized":"2023-07-14T18:24:07.407820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:19.271529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405475+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:24:07.405475+0000","last_peered":"2023-07-14T18:24:07.405475+0000","last_clean":"2023-07-14T18:24:07.405475+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:24:07.405475+0000","last_undegraded":"2023-07-14T18:24:07.405475+0000","last_fullsized":"2023-07-14T18:24:07.405475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:27:22.438148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407722+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:24:07.407722+0000","last_peered":"2023-07-14T18:24:07.407722+0000","last_clean":"2023-07-14T18:24:07.407722+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:24:07.407722+0000","last_undegraded":"2023-07-14T18:24:07.407722+0000","last_fullsized":"2023-07-14T18:24:07.407722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:18.810920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407624+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:24:07.407624+0000","last_peered":"2023-07-14T18:24:07.407624+0000","last_clean":"2023-07-14T18:24:07.407624+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:24:07.407624+0000","last_undegraded":"2023-07-14T18:24:07.407624+0000","last_fullsized":"2023-07-14T18:24:07.407624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:43.490417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406996+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:24:07.406996+0000","last_peered":"2023-07-14T18:24:07.406996+0000","last_clean":"2023-07-14T18:24:07.406996+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:24:07.406996+0000","last_undegraded":"2023-07-14T18:24:07.406996+0000","last_fullsized":"2023-07-14T18:24:07.406996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:27:08.177654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407692+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:24:07.407692+0000","last_peered":"2023-07-14T18:24:07.407692+0000","last_clean":"2023-07-14T18:24:07.407692+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:24:07.407692+0000","last_undegraded":"2023-07-14T18:24:07.407692+0000","last_fullsized":"2023-07-14T18:24:07.407692+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:37.412855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598331+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:24:07.598331+0000","last_peered":"2023-07-14T18:24:07.598331+0000","last_clean":"2023-07-14T18:24:07.598331+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:24:07.598331+0000","last_undegraded":"2023-07-14T18:24:07.598331+0000","last_fullsized":"2023-07-14T18:24:07.598331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:51.649314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409478+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:24:07.409478+0000","last_peered":"2023-07-14T18:24:07.409478+0000","last_clean":"2023-07-14T18:24:07.409478+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:24:07.409478+0000","last_undegraded":"2023-07-14T18:24:07.409478+0000","last_fullsized":"2023-07-14T18:24:07.409478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:24:08.585476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598686+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:24:07.598686+0000","last_peered":"2023-07-14T18:24:07.598686+0000","last_clean":"2023-07-14T18:24:07.598686+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:24:07.598686+0000","last_undegraded":"2023-07-14T18:24:07.598686+0000","last_fullsized":"2023-07-14T18:24:07.598686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.150320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405997+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:24:07.405997+0000","last_peered":"2023-07-14T18:24:07.405997+0000","last_clean":"2023-07-14T18:24:07.405997+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:24:07.405997+0000","last_undegraded":"2023-07-14T18:24:07.405997+0000","last_fullsized":"2023-07-14T18:24:07.405997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:37:28.657481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409648+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:24:07.409648+0000","last_peered":"2023-07-14T18:24:07.409648+0000","last_clean":"2023-07-14T18:24:07.409648+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:24:07.409648+0000","last_undegraded":"2023-07-14T18:24:07.409648+0000","last_fullsized":"2023-07-14T18:24:07.409648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:20.527639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403960+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:24:07.403960+0000","last_peered":"2023-07-14T18:24:07.403960+0000","last_clean":"2023-07-14T18:24:07.403960+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:24:07.403960+0000","last_undegraded":"2023-07-14T18:24:07.403960+0000","last_fullsized":"2023-07-14T18:24:07.403960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:43.378883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408067+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:24:07.408067+0000","last_peered":"2023-07-14T18:24:07.408067+0000","last_clean":"2023-07-14T18:24:07.408067+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:24:07.408067+0000","last_undegraded":"2023-07-14T18:24:07.408067+0000","last_fullsized":"2023-07-14T18:24:07.408067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:27.452672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409589+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:24:07.409589+0000","last_peered":"2023-07-14T18:24:07.409589+0000","last_clean":"2023-07-14T18:24:07.409589+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:24:07.409589+0000","last_undegraded":"2023-07-14T18:24:07.409589+0000","last_fullsized":"2023-07-14T18:24:07.409589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:26.524872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403362+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:24:07.403362+0000","last_peered":"2023-07-14T18:24:07.403362+0000","last_clean":"2023-07-14T18:24:07.403362+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:24:07.403362+0000","last_undegraded":"2023-07-14T18:24:07.403362+0000","last_fullsized":"2023-07-14T18:24:07.403362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:47.113574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409769+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:24:07.409769+0000","last_peered":"2023-07-14T18:24:07.409769+0000","last_clean":"2023-07-14T18:24:07.409769+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:24:07.409769+0000","last_undegraded":"2023-07-14T18:24:07.409769+0000","last_fullsized":"2023-07-14T18:24:07.409769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:07.113007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":87,"state":"activating","last_fresh":"2023-07-14T18:24:07.408312+0000","last_change":"2023-07-14T18:24:07.407003+0000","last_active":"2023-07-14T18:24:06.405140+0000","last_peered":"2023-07-14T18:23:12.490237+0000","last_clean":"2023-07-14T18:23:12.490237+0000","last_became_active":"2023-07-14T18:23:09.448108+0000","last_became_peered":"2023-07-14T18:23:09.448108+0000","last_unstale":"2023-07-14T18:24:07.408312+0000","last_undegraded":"2023-07-14T18:24:07.408312+0000","last_fullsized":"2023-07-14T18:24:07.408312+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:34.685911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7,11],"acting":[2,10,7,11],"avail_no_missing":["2(0)","5(1)","7(2)","10(1)","11(3)"],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410924+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:24:07.410924+0000","last_peered":"2023-07-14T18:24:07.410924+0000","last_clean":"2023-07-14T18:24:07.410924+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:24:07.410924+0000","last_undegraded":"2023-07-14T18:24:07.410924+0000","last_fullsized":"2023-07-14T18:24:07.410924+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:31.697182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403600+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:24:07.403600+0000","last_peered":"2023-07-14T18:24:07.403600+0000","last_clean":"2023-07-14T18:24:07.403600+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:24:07.403600+0000","last_undegraded":"2023-07-14T18:24:07.403600+0000","last_fullsized":"2023-07-14T18:24:07.403600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:44.359944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407774+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:24:07.407774+0000","last_peered":"2023-07-14T18:24:07.407774+0000","last_clean":"2023-07-14T18:24:07.407774+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:24:07.407774+0000","last_undegraded":"2023-07-14T18:24:07.407774+0000","last_fullsized":"2023-07-14T18:24:07.407774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:27.518543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.405996+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:24:06.405996+0000","last_peered":"2023-07-14T18:24:06.405996+0000","last_clean":"2023-07-14T18:24:06.405996+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:24:06.405996+0000","last_undegraded":"2023-07-14T18:24:06.405996+0000","last_fullsized":"2023-07-14T18:24:06.405996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:38:27.267958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.822326+0000","last_change":"2023-07-14T18:24:07.822326+0000","last_active":"2023-07-14T18:24:07.822326+0000","last_peered":"2023-07-14T18:24:07.822326+0000","last_clean":"2023-07-14T18:24:07.822326+0000","last_became_active":"2023-07-14T18:24:07.821961+0000","last_became_peered":"2023-07-14T18:24:07.821961+0000","last_unstale":"2023-07-14T18:24:07.822326+0000","last_undegraded":"2023-07-14T18:24:07.822326+0000","last_fullsized":"2023-07-14T18:24:07.822326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:49.138102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405293+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:24:07.405293+0000","last_peered":"2023-07-14T18:24:07.405293+0000","last_clean":"2023-07-14T18:24:07.405293+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:24:07.405293+0000","last_undegraded":"2023-07-14T18:24:07.405293+0000","last_fullsized":"2023-07-14T18:24:07.405293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:36.010004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820213+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:24:07.820213+0000","last_peered":"2023-07-14T18:24:07.820213+0000","last_clean":"2023-07-14T18:24:07.820213+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:24:07.820213+0000","last_undegraded":"2023-07-14T18:24:07.820213+0000","last_fullsized":"2023-07-14T18:24:07.820213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:32.445990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598031+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:24:07.598031+0000","last_peered":"2023-07-14T18:24:07.598031+0000","last_clean":"2023-07-14T18:24:07.598031+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:24:07.598031+0000","last_undegraded":"2023-07-14T18:24:07.598031+0000","last_fullsized":"2023-07-14T18:24:07.598031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:38.928448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405774+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:24:07.405774+0000","last_peered":"2023-07-14T18:24:07.405774+0000","last_clean":"2023-07-14T18:24:07.405774+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:24:07.405774+0000","last_undegraded":"2023-07-14T18:24:07.405774+0000","last_fullsized":"2023-07-14T18:24:07.405774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:31.966181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410048+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:24:07.410048+0000","last_peered":"2023-07-14T18:24:07.410048+0000","last_clean":"2023-07-14T18:24:07.410048+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:24:07.410048+0000","last_undegraded":"2023-07-14T18:24:07.410048+0000","last_fullsized":"2023-07-14T18:24:07.410048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:18.846973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407999+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:24:07.407999+0000","last_peered":"2023-07-14T18:24:07.407999+0000","last_clean":"2023-07-14T18:24:07.407999+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:24:07.407999+0000","last_undegraded":"2023-07-14T18:24:07.407999+0000","last_fullsized":"2023-07-14T18:24:07.407999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:36:19.820805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597965+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:24:07.597965+0000","last_peered":"2023-07-14T18:24:07.597965+0000","last_clean":"2023-07-14T18:24:07.597965+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:24:07.597965+0000","last_undegraded":"2023-07-14T18:24:07.597965+0000","last_fullsized":"2023-07-14T18:24:07.597965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:40.703098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":106,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406597+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:24:06.406597+0000","last_peered":"2023-07-14T18:24:06.406597+0000","last_clean":"2023-07-14T18:24:06.406597+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:24:06.406597+0000","last_undegraded":"2023-07-14T18:24:06.406597+0000","last_fullsized":"2023-07-14T18:24:06.406597+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:19.921934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":78,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408207+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:24:07.408207+0000","last_peered":"2023-07-14T18:24:07.408207+0000","last_clean":"2023-07-14T18:24:07.408207+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:24:07.408207+0000","last_undegraded":"2023-07-14T18:24:07.408207+0000","last_fullsized":"2023-07-14T18:24:07.408207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:44:26.421835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413138+0000","last_change":"2023-07-14T18:24:07.412404+0000","last_active":"2023-07-14T18:24:07.413138+0000","last_peered":"2023-07-14T18:24:07.413138+0000","last_clean":"2023-07-14T18:24:07.413138+0000","last_became_active":"2023-07-14T18:24:07.411933+0000","last_became_peered":"2023-07-14T18:24:07.411933+0000","last_unstale":"2023-07-14T18:24:07.413138+0000","last_undegraded":"2023-07-14T18:24:07.413138+0000","last_fullsized":"2023-07-14T18:24:07.413138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:22.236815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406710+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:24:07.406710+0000","last_peered":"2023-07-14T18:24:07.406710+0000","last_clean":"2023-07-14T18:24:07.406710+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:24:07.406710+0000","last_undegraded":"2023-07-14T18:24:07.406710+0000","last_fullsized":"2023-07-14T18:24:07.406710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:32:56.571997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.411973+0000","last_change":"2023-07-14T18:24:07.411973+0000","last_active":"2023-07-14T18:24:07.411973+0000","last_peered":"2023-07-14T18:24:07.411973+0000","last_clean":"2023-07-14T18:24:07.411973+0000","last_became_active":"2023-07-14T18:24:07.411638+0000","last_became_peered":"2023-07-14T18:24:07.411638+0000","last_unstale":"2023-07-14T18:24:07.411973+0000","last_undegraded":"2023-07-14T18:24:07.411973+0000","last_fullsized":"2023-07-14T18:24:07.411973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:28.531130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413964+0000","last_change":"2023-07-14T18:24:07.413964+0000","last_active":"2023-07-14T18:24:07.413964+0000","last_peered":"2023-07-14T18:24:07.413964+0000","last_clean":"2023-07-14T18:24:07.413964+0000","last_became_active":"2023-07-14T18:24:07.413507+0000","last_became_peered":"2023-07-14T18:24:07.413507+0000","last_unstale":"2023-07-14T18:24:07.413964+0000","last_undegraded":"2023-07-14T18:24:07.413964+0000","last_fullsized":"2023-07-14T18:24:07.413964+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:27.116037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409637+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:24:07.409637+0000","last_peered":"2023-07-14T18:24:07.409637+0000","last_clean":"2023-07-14T18:24:07.409637+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:24:07.409637+0000","last_undegraded":"2023-07-14T18:24:07.409637+0000","last_fullsized":"2023-07-14T18:24:07.409637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:25.197615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408340+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:24:07.408340+0000","last_peered":"2023-07-14T18:24:07.408340+0000","last_clean":"2023-07-14T18:24:07.408340+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:24:07.408340+0000","last_undegraded":"2023-07-14T18:24:07.408340+0000","last_fullsized":"2023-07-14T18:24:07.408340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:15:33.259444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407780+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:24:07.407780+0000","last_peered":"2023-07-14T18:24:07.407780+0000","last_clean":"2023-07-14T18:24:07.407780+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:24:07.407780+0000","last_undegraded":"2023-07-14T18:24:07.407780+0000","last_fullsized":"2023-07-14T18:24:07.407780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:42.309537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'33","reported_seq":141,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406505+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:24:06.406505+0000","last_peered":"2023-07-14T18:24:06.406505+0000","last_clean":"2023-07-14T18:24:06.406505+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:24:06.406505+0000","last_undegraded":"2023-07-14T18:24:06.406505+0000","last_fullsized":"2023-07-14T18:24:06.406505+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:10.210991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":84,"num_read_kb":98,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403773+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:24:07.403773+0000","last_peered":"2023-07-14T18:24:07.403773+0000","last_clean":"2023-07-14T18:24:07.403773+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:24:07.403773+0000","last_undegraded":"2023-07-14T18:24:07.403773+0000","last_fullsized":"2023-07-14T18:24:07.403773+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:19.677064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410030+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:24:07.410030+0000","last_peered":"2023-07-14T18:24:07.410030+0000","last_clean":"2023-07-14T18:24:07.410030+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:24:07.410030+0000","last_undegraded":"2023-07-14T18:24:07.410030+0000","last_fullsized":"2023-07-14T18:24:07.410030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:15.831940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407198+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:24:07.407198+0000","last_peered":"2023-07-14T18:24:07.407198+0000","last_clean":"2023-07-14T18:24:07.407198+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:24:07.407198+0000","last_undegraded":"2023-07-14T18:24:07.407198+0000","last_fullsized":"2023-07-14T18:24:07.407198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:03.898501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820320+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:24:07.820320+0000","last_peered":"2023-07-14T18:24:07.820320+0000","last_clean":"2023-07-14T18:24:07.820320+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:24:07.820320+0000","last_undegraded":"2023-07-14T18:24:07.820320+0000","last_fullsized":"2023-07-14T18:24:07.820320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:28:38.028092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405119+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:24:07.405119+0000","last_peered":"2023-07-14T18:24:07.405119+0000","last_clean":"2023-07-14T18:24:07.405119+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:24:07.405119+0000","last_undegraded":"2023-07-14T18:24:07.405119+0000","last_fullsized":"2023-07-14T18:24:07.405119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:57.627580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409613+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:24:07.409613+0000","last_peered":"2023-07-14T18:24:07.409613+0000","last_clean":"2023-07-14T18:24:07.409613+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:24:07.409613+0000","last_undegraded":"2023-07-14T18:24:07.409613+0000","last_fullsized":"2023-07-14T18:24:07.409613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:13.349187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405811+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:24:07.405811+0000","last_peered":"2023-07-14T18:24:07.405811+0000","last_clean":"2023-07-14T18:24:07.405811+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:24:07.405811+0000","last_undegraded":"2023-07-14T18:24:07.405811+0000","last_fullsized":"2023-07-14T18:24:07.405811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:17:48.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407367+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:24:07.407367+0000","last_peered":"2023-07-14T18:24:07.407367+0000","last_clean":"2023-07-14T18:24:07.407367+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:24:07.407367+0000","last_undegraded":"2023-07-14T18:24:07.407367+0000","last_fullsized":"2023-07-14T18:24:07.407367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:31:02.627246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:08.356803+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:24:08.356803+0000","last_peered":"2023-07-14T18:24:08.356803+0000","last_clean":"2023-07-14T18:24:08.356803+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:24:08.356803+0000","last_undegraded":"2023-07-14T18:24:08.356803+0000","last_fullsized":"2023-07-14T18:24:08.356803+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:05.604630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405854+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:24:07.405854+0000","last_peered":"2023-07-14T18:24:07.405854+0000","last_clean":"2023-07-14T18:24:07.405854+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:24:07.405854+0000","last_undegraded":"2023-07-14T18:24:07.405854+0000","last_fullsized":"2023-07-14T18:24:07.405854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.422448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820996+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:24:07.820996+0000","last_peered":"2023-07-14T18:24:07.820996+0000","last_clean":"2023-07-14T18:24:07.820996+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:24:07.820996+0000","last_undegraded":"2023-07-14T18:24:07.820996+0000","last_fullsized":"2023-07-14T18:24:07.820996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:00.885274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403130+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:24:07.403130+0000","last_peered":"2023-07-14T18:24:07.403130+0000","last_clean":"2023-07-14T18:24:07.403130+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:24:07.403130+0000","last_undegraded":"2023-07-14T18:24:07.403130+0000","last_fullsized":"2023-07-14T18:24:07.403130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:22.401795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413544+0000","last_change":"2023-07-14T18:24:07.413544+0000","last_active":"2023-07-14T18:24:07.413544+0000","last_peered":"2023-07-14T18:24:07.413544+0000","last_clean":"2023-07-14T18:24:07.413544+0000","last_became_active":"2023-07-14T18:24:07.413217+0000","last_became_peered":"2023-07-14T18:24:07.413217+0000","last_unstale":"2023-07-14T18:24:07.413544+0000","last_undegraded":"2023-07-14T18:24:07.413544+0000","last_fullsized":"2023-07-14T18:24:07.413544+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:24.792524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410040+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:24:07.410040+0000","last_peered":"2023-07-14T18:24:07.410040+0000","last_clean":"2023-07-14T18:24:07.410040+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:24:07.410040+0000","last_undegraded":"2023-07-14T18:24:07.410040+0000","last_fullsized":"2023-07-14T18:24:07.410040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:18.284526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404072+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:24:07.404072+0000","last_peered":"2023-07-14T18:24:07.404072+0000","last_clean":"2023-07-14T18:24:07.404072+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:24:07.404072+0000","last_undegraded":"2023-07-14T18:24:07.404072+0000","last_fullsized":"2023-07-14T18:24:07.404072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:44.906119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405246+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:24:07.405246+0000","last_peered":"2023-07-14T18:24:07.405246+0000","last_clean":"2023-07-14T18:24:07.405246+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:24:07.405246+0000","last_undegraded":"2023-07-14T18:24:07.405246+0000","last_fullsized":"2023-07-14T18:24:07.405246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:41.007811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403367+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:24:07.403367+0000","last_peered":"2023-07-14T18:24:07.403367+0000","last_clean":"2023-07-14T18:24:07.403367+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:24:07.403367+0000","last_undegraded":"2023-07-14T18:24:07.403367+0000","last_fullsized":"2023-07-14T18:24:07.403367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:06.743747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413261+0000","last_change":"2023-07-14T18:24:07.413261+0000","last_active":"2023-07-14T18:24:07.413261+0000","last_peered":"2023-07-14T18:24:07.413261+0000","last_clean":"2023-07-14T18:24:07.413261+0000","last_became_active":"2023-07-14T18:24:07.412759+0000","last_became_peered":"2023-07-14T18:24:07.412759+0000","last_unstale":"2023-07-14T18:24:07.413261+0000","last_undegraded":"2023-07-14T18:24:07.413261+0000","last_fullsized":"2023-07-14T18:24:07.413261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:09.725334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406205+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:24:06.406205+0000","last_peered":"2023-07-14T18:24:06.406205+0000","last_clean":"2023-07-14T18:24:06.406205+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:24:06.406205+0000","last_undegraded":"2023-07-14T18:24:06.406205+0000","last_fullsized":"2023-07-14T18:24:06.406205+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:00.302894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820487+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:24:07.820487+0000","last_peered":"2023-07-14T18:24:07.820487+0000","last_clean":"2023-07-14T18:24:07.820487+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:24:07.820487+0000","last_undegraded":"2023-07-14T18:24:07.820487+0000","last_fullsized":"2023-07-14T18:24:07.820487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:09:47.913784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405859+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:24:07.405859+0000","last_peered":"2023-07-14T18:24:07.405859+0000","last_clean":"2023-07-14T18:24:07.405859+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:24:07.405859+0000","last_undegraded":"2023-07-14T18:24:07.405859+0000","last_fullsized":"2023-07-14T18:24:07.405859+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:23.957543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":42,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T18:24:06.406294+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:24:06.406294+0000","last_peered":"2023-07-14T18:24:06.406294+0000","last_clean":"2023-07-14T18:24:06.406294+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:24:06.406294+0000","last_undegraded":"2023-07-14T18:24:06.406294+0000","last_fullsized":"2023-07-14T18:24:06.406294+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:13:32.953610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821088+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:24:07.821088+0000","last_peered":"2023-07-14T18:24:07.821088+0000","last_clean":"2023-07-14T18:24:07.821088+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:24:07.821088+0000","last_undegraded":"2023-07-14T18:24:07.821088+0000","last_fullsized":"2023-07-14T18:24:07.821088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:26.669137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410319+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:24:07.410319+0000","last_peered":"2023-07-14T18:24:07.410319+0000","last_clean":"2023-07-14T18:24:07.410319+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:24:07.410319+0000","last_undegraded":"2023-07-14T18:24:07.410319+0000","last_fullsized":"2023-07-14T18:24:07.410319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:11.487403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":74,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:08.355867+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:24:08.355867+0000","last_peered":"2023-07-14T18:24:08.355867+0000","last_clean":"2023-07-14T18:24:08.355867+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:24:08.355867+0000","last_undegraded":"2023-07-14T18:24:08.355867+0000","last_fullsized":"2023-07-14T18:24:08.355867+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:44.654417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406370+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:24:07.406370+0000","last_peered":"2023-07-14T18:24:07.406370+0000","last_clean":"2023-07-14T18:24:07.406370+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:24:07.406370+0000","last_undegraded":"2023-07-14T18:24:07.406370+0000","last_fullsized":"2023-07-14T18:24:07.406370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:01.561467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:01.628715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409780+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:24:07.409780+0000","last_peered":"2023-07-14T18:24:07.409780+0000","last_clean":"2023-07-14T18:24:07.409780+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:24:07.409780+0000","last_undegraded":"2023-07-14T18:24:07.409780+0000","last_fullsized":"2023-07-14T18:24:07.409780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:45:44.875754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405706+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:24:07.405706+0000","last_peered":"2023-07-14T18:24:07.405706+0000","last_clean":"2023-07-14T18:24:07.405706+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:24:07.405706+0000","last_undegraded":"2023-07-14T18:24:07.405706+0000","last_fullsized":"2023-07-14T18:24:07.405706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:07.000585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410311+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:24:07.410311+0000","last_peered":"2023-07-14T18:24:07.410311+0000","last_clean":"2023-07-14T18:24:07.410311+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:24:07.410311+0000","last_undegraded":"2023-07-14T18:24:07.410311+0000","last_fullsized":"2023-07-14T18:24:07.410311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:22.806069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820058+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:24:07.820058+0000","last_peered":"2023-07-14T18:24:07.820058+0000","last_clean":"2023-07-14T18:24:07.820058+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:24:07.820058+0000","last_undegraded":"2023-07-14T18:24:07.820058+0000","last_fullsized":"2023-07-14T18:24:07.820058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:00.275183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405700+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:24:07.405700+0000","last_peered":"2023-07-14T18:24:07.405700+0000","last_clean":"2023-07-14T18:24:07.405700+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:24:07.405700+0000","last_undegraded":"2023-07-14T18:24:07.405700+0000","last_fullsized":"2023-07-14T18:24:07.405700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:04.719205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405561+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:24:07.405561+0000","last_peered":"2023-07-14T18:24:07.405561+0000","last_clean":"2023-07-14T18:24:07.405561+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:24:07.405561+0000","last_undegraded":"2023-07-14T18:24:07.405561+0000","last_fullsized":"2023-07-14T18:24:07.405561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:44.840479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403787+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:24:07.403787+0000","last_peered":"2023-07-14T18:24:07.403787+0000","last_clean":"2023-07-14T18:24:07.403787+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:24:07.403787+0000","last_undegraded":"2023-07-14T18:24:07.403787+0000","last_fullsized":"2023-07-14T18:24:07.403787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.485409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820701+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:24:07.820701+0000","last_peered":"2023-07-14T18:24:07.820701+0000","last_clean":"2023-07-14T18:24:07.820701+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:24:07.820701+0000","last_undegraded":"2023-07-14T18:24:07.820701+0000","last_fullsized":"2023-07-14T18:24:07.820701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:52:09.481241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597956+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:24:07.597956+0000","last_peered":"2023-07-14T18:24:07.597956+0000","last_clean":"2023-07-14T18:24:07.597956+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:24:07.597956+0000","last_undegraded":"2023-07-14T18:24:07.597956+0000","last_fullsized":"2023-07-14T18:24:07.597956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:55.573942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407882+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:24:07.407882+0000","last_peered":"2023-07-14T18:24:07.407882+0000","last_clean":"2023-07-14T18:24:07.407882+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:24:07.407882+0000","last_undegraded":"2023-07-14T18:24:07.407882+0000","last_fullsized":"2023-07-14T18:24:07.407882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:55:27.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598511+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:24:07.598511+0000","last_peered":"2023-07-14T18:24:07.598511+0000","last_clean":"2023-07-14T18:24:07.598511+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:24:07.598511+0000","last_undegraded":"2023-07-14T18:24:07.598511+0000","last_fullsized":"2023-07-14T18:24:07.598511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:57.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406745+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:24:07.406745+0000","last_peered":"2023-07-14T18:24:07.406745+0000","last_clean":"2023-07-14T18:24:07.406745+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:24:07.406745+0000","last_undegraded":"2023-07-14T18:24:07.406745+0000","last_fullsized":"2023-07-14T18:24:07.406745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:40.891887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410518+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:24:07.410518+0000","last_peered":"2023-07-14T18:24:07.410518+0000","last_clean":"2023-07-14T18:24:07.410518+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:24:07.410518+0000","last_undegraded":"2023-07-14T18:24:07.410518+0000","last_fullsized":"2023-07-14T18:24:07.410518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:54.251569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405928+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:24:07.405928+0000","last_peered":"2023-07-14T18:24:07.405928+0000","last_clean":"2023-07-14T18:24:07.405928+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:24:07.405928+0000","last_undegraded":"2023-07-14T18:24:07.405928+0000","last_fullsized":"2023-07-14T18:24:07.405928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:46.633579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599735+0000","last_change":"2023-07-14T18:24:07.599735+0000","last_active":"2023-07-14T18:24:07.599735+0000","last_peered":"2023-07-14T18:24:07.599735+0000","last_clean":"2023-07-14T18:24:07.599735+0000","last_became_active":"2023-07-14T18:24:07.599353+0000","last_became_peered":"2023-07-14T18:24:07.599353+0000","last_unstale":"2023-07-14T18:24:07.599735+0000","last_undegraded":"2023-07-14T18:24:07.599735+0000","last_fullsized":"2023-07-14T18:24:07.599735+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:36.613425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407843+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:24:07.407843+0000","last_peered":"2023-07-14T18:24:07.407843+0000","last_clean":"2023-07-14T18:24:07.407843+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:24:07.407843+0000","last_undegraded":"2023-07-14T18:24:07.407843+0000","last_fullsized":"2023-07-14T18:24:07.407843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:27:45.050723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403163+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:24:07.403163+0000","last_peered":"2023-07-14T18:24:07.403163+0000","last_clean":"2023-07-14T18:24:07.403163+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:24:07.403163+0000","last_undegraded":"2023-07-14T18:24:07.403163+0000","last_fullsized":"2023-07-14T18:24:07.403163+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:15:37.710301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407589+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:24:07.407589+0000","last_peered":"2023-07-14T18:24:07.407589+0000","last_clean":"2023-07-14T18:24:07.407589+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:24:07.407589+0000","last_undegraded":"2023-07-14T18:24:07.407589+0000","last_fullsized":"2023-07-14T18:24:07.407589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:02.479091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:37.977157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404952+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:24:07.404952+0000","last_peered":"2023-07-14T18:24:07.404952+0000","last_clean":"2023-07-14T18:24:07.404952+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:24:07.404952+0000","last_undegraded":"2023-07-14T18:24:07.404952+0000","last_fullsized":"2023-07-14T18:24:07.404952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:37.080797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410069+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:24:07.410069+0000","last_peered":"2023-07-14T18:24:07.410069+0000","last_clean":"2023-07-14T18:24:07.410069+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:24:07.410069+0000","last_undegraded":"2023-07-14T18:24:07.410069+0000","last_fullsized":"2023-07-14T18:24:07.410069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:00.967275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407086+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:24:07.407086+0000","last_peered":"2023-07-14T18:24:07.407086+0000","last_clean":"2023-07-14T18:24:07.407086+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:24:07.407086+0000","last_undegraded":"2023-07-14T18:24:07.407086+0000","last_fullsized":"2023-07-14T18:24:07.407086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:39.830952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598357+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:24:07.598357+0000","last_peered":"2023-07-14T18:24:07.598357+0000","last_clean":"2023-07-14T18:24:07.598357+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:24:07.598357+0000","last_undegraded":"2023-07-14T18:24:07.598357+0000","last_fullsized":"2023-07-14T18:24:07.598357+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:12.153021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409937+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:24:07.409937+0000","last_peered":"2023-07-14T18:24:07.409937+0000","last_clean":"2023-07-14T18:24:07.409937+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:24:07.409937+0000","last_undegraded":"2023-07-14T18:24:07.409937+0000","last_fullsized":"2023-07-14T18:24:07.409937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:41.469044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.416915+0000","last_change":"2023-07-14T18:24:07.416556+0000","last_active":"2023-07-14T18:24:07.416915+0000","last_peered":"2023-07-14T18:24:07.416915+0000","last_clean":"2023-07-14T18:24:07.416915+0000","last_became_active":"2023-07-14T18:24:07.408596+0000","last_became_peered":"2023-07-14T18:24:07.408596+0000","last_unstale":"2023-07-14T18:24:07.416915+0000","last_undegraded":"2023-07-14T18:24:07.416915+0000","last_fullsized":"2023-07-14T18:24:07.416915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:10.303257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405892+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:24:07.405892+0000","last_peered":"2023-07-14T18:24:07.405892+0000","last_clean":"2023-07-14T18:24:07.405892+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:24:07.405892+0000","last_undegraded":"2023-07-14T18:24:07.405892+0000","last_fullsized":"2023-07-14T18:24:07.405892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:59:09.124887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407944+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:24:07.407944+0000","last_peered":"2023-07-14T18:24:07.407944+0000","last_clean":"2023-07-14T18:24:07.407944+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:24:07.407944+0000","last_undegraded":"2023-07-14T18:24:07.407944+0000","last_fullsized":"2023-07-14T18:24:07.407944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:33.118185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407530+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:24:07.407530+0000","last_peered":"2023-07-14T18:24:07.407530+0000","last_clean":"2023-07-14T18:24:07.407530+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:24:07.407530+0000","last_undegraded":"2023-07-14T18:24:07.407530+0000","last_fullsized":"2023-07-14T18:24:07.407530+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:33.188924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406610+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:24:07.406610+0000","last_peered":"2023-07-14T18:24:07.406610+0000","last_clean":"2023-07-14T18:24:07.406610+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:24:07.406610+0000","last_undegraded":"2023-07-14T18:24:07.406610+0000","last_fullsized":"2023-07-14T18:24:07.406610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:25:28.457835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406508+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:24:07.406508+0000","last_peered":"2023-07-14T18:24:07.406508+0000","last_clean":"2023-07-14T18:24:07.406508+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:24:07.406508+0000","last_undegraded":"2023-07-14T18:24:07.406508+0000","last_fullsized":"2023-07-14T18:24:07.406508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:42:15.619889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409875+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:24:07.409875+0000","last_peered":"2023-07-14T18:24:07.409875+0000","last_clean":"2023-07-14T18:24:07.409875+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:24:07.409875+0000","last_undegraded":"2023-07-14T18:24:07.409875+0000","last_fullsized":"2023-07-14T18:24:07.409875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:21:05.734566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406230+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:24:07.406230+0000","last_peered":"2023-07-14T18:24:07.406230+0000","last_clean":"2023-07-14T18:24:07.406230+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:24:07.406230+0000","last_undegraded":"2023-07-14T18:24:07.406230+0000","last_fullsized":"2023-07-14T18:24:07.406230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:18.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408921+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:24:07.408921+0000","last_peered":"2023-07-14T18:24:07.408921+0000","last_clean":"2023-07-14T18:24:07.408921+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:24:07.408921+0000","last_undegraded":"2023-07-14T18:24:07.408921+0000","last_fullsized":"2023-07-14T18:24:07.408921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:45.492125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821139+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:24:07.821139+0000","last_peered":"2023-07-14T18:24:07.821139+0000","last_clean":"2023-07-14T18:24:07.821139+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:24:07.821139+0000","last_undegraded":"2023-07-14T18:24:07.821139+0000","last_fullsized":"2023-07-14T18:24:07.821139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:41.844783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407181+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:24:07.407181+0000","last_peered":"2023-07-14T18:24:07.407181+0000","last_clean":"2023-07-14T18:24:07.407181+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:24:07.407181+0000","last_undegraded":"2023-07-14T18:24:07.407181+0000","last_fullsized":"2023-07-14T18:24:07.407181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:34:35.248238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405025+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:24:07.405025+0000","last_peered":"2023-07-14T18:24:07.405025+0000","last_clean":"2023-07-14T18:24:07.405025+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:24:07.405025+0000","last_undegraded":"2023-07-14T18:24:07.405025+0000","last_fullsized":"2023-07-14T18:24:07.405025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:53.390475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406959+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:24:07.406959+0000","last_peered":"2023-07-14T18:24:07.406959+0000","last_clean":"2023-07-14T18:24:07.406959+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:24:07.406959+0000","last_undegraded":"2023-07-14T18:24:07.406959+0000","last_fullsized":"2023-07-14T18:24:07.406959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:26.096564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406915+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:24:07.406915+0000","last_peered":"2023-07-14T18:24:07.406915+0000","last_clean":"2023-07-14T18:24:07.406915+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:24:07.406915+0000","last_undegraded":"2023-07-14T18:24:07.406915+0000","last_fullsized":"2023-07-14T18:24:07.406915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:32:24.873296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406046+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:24:07.406046+0000","last_peered":"2023-07-14T18:24:07.406046+0000","last_clean":"2023-07-14T18:24:07.406046+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:24:07.406046+0000","last_undegraded":"2023-07-14T18:24:07.406046+0000","last_fullsized":"2023-07-14T18:24:07.406046+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:58.884740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820615+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:24:07.820615+0000","last_peered":"2023-07-14T18:24:07.820615+0000","last_clean":"2023-07-14T18:24:07.820615+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:24:07.820615+0000","last_undegraded":"2023-07-14T18:24:07.820615+0000","last_fullsized":"2023-07-14T18:24:07.820615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:23:26.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405289+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:24:07.405289+0000","last_peered":"2023-07-14T18:24:07.405289+0000","last_clean":"2023-07-14T18:24:07.405289+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:24:07.405289+0000","last_undegraded":"2023-07-14T18:24:07.405289+0000","last_fullsized":"2023-07-14T18:24:07.405289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:02.717582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407277+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:24:07.407277+0000","last_peered":"2023-07-14T18:24:07.407277+0000","last_clean":"2023-07-14T18:24:07.407277+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:24:07.407277+0000","last_undegraded":"2023-07-14T18:24:07.407277+0000","last_fullsized":"2023-07-14T18:24:07.407277+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:15:50.630547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598631+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:24:07.598631+0000","last_peered":"2023-07-14T18:24:07.598631+0000","last_clean":"2023-07-14T18:24:07.598631+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:24:07.598631+0000","last_undegraded":"2023-07-14T18:24:07.598631+0000","last_fullsized":"2023-07-14T18:24:07.598631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:06.982685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410189+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:24:07.410189+0000","last_peered":"2023-07-14T18:24:07.410189+0000","last_clean":"2023-07-14T18:24:07.410189+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:24:07.410189+0000","last_undegraded":"2023-07-14T18:24:07.410189+0000","last_fullsized":"2023-07-14T18:24:07.410189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:49:31.338811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598143+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:24:07.598143+0000","last_peered":"2023-07-14T18:24:07.598143+0000","last_clean":"2023-07-14T18:24:07.598143+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:24:07.598143+0000","last_undegraded":"2023-07-14T18:24:07.598143+0000","last_fullsized":"2023-07-14T18:24:07.598143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408224+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:24:07.408224+0000","last_peered":"2023-07-14T18:24:07.408224+0000","last_clean":"2023-07-14T18:24:07.408224+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:24:07.408224+0000","last_undegraded":"2023-07-14T18:24:07.408224+0000","last_fullsized":"2023-07-14T18:24:07.408224+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.457179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820178+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:24:07.820178+0000","last_peered":"2023-07-14T18:24:07.820178+0000","last_clean":"2023-07-14T18:24:07.820178+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:24:07.820178+0000","last_undegraded":"2023-07-14T18:24:07.820178+0000","last_fullsized":"2023-07-14T18:24:07.820178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:24.150124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405223+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:24:07.405223+0000","last_peered":"2023-07-14T18:24:07.405223+0000","last_clean":"2023-07-14T18:24:07.405223+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:24:07.405223+0000","last_undegraded":"2023-07-14T18:24:07.405223+0000","last_fullsized":"2023-07-14T18:24:07.405223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:22.497297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":435,"num_read_kb":286,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514520,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":477638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53800000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52300000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56000000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.64400000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.66200000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.625},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68400000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.72899999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.74099999999999999}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28616,"kb_used_data":1600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716536,"statfs":{"total":95995035648,"available":95965732864,"internally_reserved":0,"allocated":1638400,"data_stored":1103531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.53000000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.57299999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63,"15min":0.63},"min":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.78600000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.624},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64300000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.497}]},{"osd":4,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.433},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.58699999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.82999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.872,"5min":1.052,"15min":1.052},"last":0.67800000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.58099999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.68300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.92600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.80200000000000005},"min":{"1min":0.64600000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78800000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.57599999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.87,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84699999999999998,"5min":1.256,"15min":1.256},"last":0.72499999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.75800000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":991232,"data_stored":471748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.499,"15min":0.46400000000000002},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":5.8559999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.874,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.5140000000000002,"5min":5.5140000000000002,"15min":5.5140000000000002},"last":0.624}]},{"osd":2,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.71299999999999997,"15min":0.68799999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.641,"5min":5.641,"15min":5.641},"last":0.35099999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.318}]},{"osd":4,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5679999999999996,"5min":5.5679999999999996,"15min":5.5679999999999996},"last":0.745},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.59699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78200000000000003,"15min":0.79400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.41999999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80400000000000005,"15min":0.81399999999999995},"min":{"1min":0.53000000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.68799999999999994}]},{"osd":8,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.85299999999999998,"15min":0.82899999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.875},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.53100000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.57499999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":477604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.443}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.1280000000000001,"5min":3.1280000000000001,"15min":3.1280000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.625}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.59599999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.48299999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64800000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70399999999999996,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.55200000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.71899999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68000000000000005}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69699999999999995}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1073683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.371},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.56999999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.46400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68799999999999994}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.86899999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.38200000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.441}]},{"osd":5,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89600000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.753},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.86199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59799999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92500000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90500000000000003,"5min":1.02,"15min":1.02},"last":0.624}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.77300000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97099999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.66600000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:23:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84499999999999997}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72599999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":472409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40500000000000003},"min":{"1min":0.28299999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.60299999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93300000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.45800000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96399999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.61099999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.63},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.80100000000000005,"5min":1.099,"15min":1.099},"last":0.501}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.56599999999999995}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.621}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76000000000000001,"15min":0.76500000000000001},"min":{"1min":0.63,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.64000000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":532263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.441},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.53400000000000003},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.42999999999999999},"min":{"1min":0.35799999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71199999999999997,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.48499999999999999,"15min":0.46200000000000002},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.76700000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.497}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.64000000000000001,"15min":0.61299999999999999},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93999999999999995,"5min":1.089,"15min":1.089},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.64000000000000001,"15min":0.61499999999999999},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63900000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.498,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0249999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.623,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88900000000000001,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55400000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69499999999999995,"15min":0.67900000000000005},"min":{"1min":0.54600000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.012,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.53100000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":1.173,"15min":1.173},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.097,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.63},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.65900000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.62},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":474255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56899999999999995,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81100000000000005,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.96599999999999997,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.438}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.66900000000000004}]},{"osd":4,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.54700000000000004},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.61499999999999999,"15min":0.57099999999999995},"min":{"1min":0.498,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.50700000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.48599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89800000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.56100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97399999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.58299999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.78500000000000003},"min":{"1min":0.503,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.85499999999999998,"5min":1.036,"15min":1.036},"last":0.75900000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.442,"15min":0.442},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.496},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.65500000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91400000000000003,"5min":1.075,"15min":1.075},"last":0.69699999999999995}]},{"osd":11,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.65200000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":523641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.374},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.54100000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.497},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.434}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.45200000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.72299999999999998,"15min":0.70499999999999996},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.58399999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.621,"15min":1.621},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.627}]},{"osd":11,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.54700000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":472157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41199999999999998,"15min":0.374},"min":{"1min":0.29099999999999998,"5min":0.161,"15min":0.161},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.47799999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.629,"15min":0.57899999999999996},"min":{"1min":0.41799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":8.1820000000000004},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":1.0289999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.68200000000000005,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.2210000000000001,"5min":8.2210000000000001,"15min":8.2210000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.93000000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":1.0049999999999999}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.68500000000000005,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":1.1259999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.69199999999999995,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.81000000000000005,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":11.609999999999999,"5min":11.609999999999999,"15min":11.609999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.125,"5min":0.84499999999999997,"15min":0.79900000000000004},"min":{"1min":0.497,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.0559999999999992,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":1.044}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.28,"5min":0.879,"15min":0.81200000000000006},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.83699999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.90400000000000003,"15min":0.83799999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":12.291,"5min":12.291,"15min":12.291},"last":1.0629999999999999}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":11.263999999999999,"5min":11.263999999999999,"15min":11.263999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.1060000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":476483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.37},"min":{"1min":0.29999999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54400000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.317},{"interface":"front","average":{"1min":0.437,"5min":0.41899999999999998,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.621,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34100000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.57499999999999996,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.56200000000000006},"min":{"1min":0.495,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.48399999999999999}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56100000000000005,"15min":0.53000000000000003},"min":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78300000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.55900000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.503},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.51500000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89000000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.57899999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.60499999999999998},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.44700000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39800000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.54800000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.753}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.74199999999999999,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.98099999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.495},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.53400000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59199999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1601536,"data_stored":1069303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.34799999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.41099999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.307},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.32500000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.42299999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.45400000000000001}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.434},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83399999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.52100000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68000000000000005,"15min":0.66000000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94499999999999995,"5min":1.075,"15min":1.075},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84699999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.47999999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.498}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89600000000000002,"5min":1.087,"15min":1.087},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.80200000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.98799999999999999,"5min":1.206,"15min":1.206},"last":0.57699999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95699999999999996,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77100000000000002},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.873,"5min":1.175,"15min":1.175},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:24:12.862 INFO:tasks.cephadm.ceph_manager.ceph:PG 4.1d is not active+clean 2023-07-14T18:24:12.862 INFO:tasks.cephadm.ceph_manager.ceph:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 87, 'state': 'activating', 'last_fresh': '2023-07-14T18:24:07.408312+0000', 'last_change': '2023-07-14T18:24:07.407003+0000', 'last_active': '2023-07-14T18:24:06.405140+0000', 'last_peered': '2023-07-14T18:23:12.490237+0000', 'last_clean': '2023-07-14T18:23:12.490237+0000', 'last_became_active': '2023-07-14T18:23:09.448108+0000', 'last_became_peered': '2023-07-14T18:23:09.448108+0000', 'last_unstale': '2023-07-14T18:24:07.408312+0000', 'last_undegraded': '2023-07-14T18:24:07.408312+0000', 'last_fullsized': '2023-07-14T18:24:07.408312+0000', 'mapping_epoch': 86, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 83, 'last_epoch_clean': 84, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-07-14T18:23:08.423883+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-07-14T18:23:08.423883+0000', 'last_clean_scrub_stamp': '2023-07-14T18:23:08.423883+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-07-15T23:49:34.685911+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 10, 7, 11], 'acting': [2, 10, 7, 11], 'avail_no_missing': ['2(0)', '5(1)', '7(2)', '10(1)', '11(3)'], 'object_location_counts': [], 'blocked_by': [1], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-07-14T18:24:12.863 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph status --format=json 2023-07-14T18:24:13.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:12 smithi105 ceph-mon[98013]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:13.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:12 smithi105 ceph-mon[98013]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:13.092 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:12 smithi064 ceph-mon[96720]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:13.092 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:12 smithi064 ceph-mon[96720]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:13.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:12 smithi144 ceph-mon[98417]: pgmap v290: 129 pgs: 1 activating, 128 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:13.183 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:12 smithi144 ceph-mon[98417]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:13.421 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- 172.21.15.64:0/858615970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400ff7e0 msgr2=0x7f18401037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 --2- 172.21.15.64:0/858615970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400ff7e0 0x7f18401037b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f18300099c0 tx=0x7f1830009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- 172.21.15.64:0/858615970 shutdown_connections 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 --2- 172.21.15.64:0/858615970 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400fee40 0x7f18400ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 --2- 172.21.15.64:0/858615970 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1840103ee0 0x7f184010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 --2- 172.21.15.64:0/858615970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f18400ff7e0 0x7f18401037b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- 172.21.15.64:0/858615970 >> 172.21.15.64:0/858615970 conn(0x7f1840075ca0 msgr2=0x7f18400760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- 172.21.15.64:0/858615970 shutdown_connections 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- 172.21.15.64:0/858615970 wait complete. 2023-07-14T18:24:14.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 Processor -- start 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.532+0000 7f1847512700 1 -- start start 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18400fee40 0x7f1840100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400ff7e0 0x7f1840100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401a41d0 con 0x7f1840103ee0 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401a4340 con 0x7f18400fee40 2023-07-14T18:24:14.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1847512700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18401a4500 con 0x7f18400ff7e0 2023-07-14T18:24:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47922/0 (socket says 172.21.15.64:47922) 2023-07-14T18:24:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1844aad700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400ff7e0 0x7f1840100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 -- 172.21.15.64:0/2720406419 learned_addr learned my addr 172.21.15.64:0/2720406419 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:14.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f18452ae700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18400fee40 0x7f1840100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 -- 172.21.15.64:0/2720406419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400ff7e0 msgr2=0x7f1840100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400ff7e0 0x7f1840100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 -- 172.21.15.64:0/2720406419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18400fee40 msgr2=0x7f1840100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18400fee40 0x7f1840100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 -- 172.21.15.64:0/2720406419 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300095f0 con 0x7f1840103ee0 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f1845aaf700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1838007f10 tx=0x7f183800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:14.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.533+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838017070 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.534+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18401a47e0 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.534+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f183800f040 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.534+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838005570 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.534+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18401a4c60 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.534+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18401081c0 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.536+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f183800db20 con 0x7f1840103ee0 2023-07-14T18:24:14.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.537+0000 7f18367fc700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 0x7f182c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:14.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.537+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f18380cca90 con 0x7f1840103ee0 2023-07-14T18:24:14.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.537+0000 7f18452ae700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 0x7f182c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:14.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.537+0000 7f18452ae700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 0x7f182c0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f183c005950 tx=0x7f183c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:14.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.540+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1838095d70 con 0x7f1840103ee0 2023-07-14T18:24:14.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.817+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f184004e340 con 0x7f1840103ee0 2023-07-14T18:24:14.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.817+0000 7f18367fc700 1 -- 172.21.15.64:0/2720406419 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+3136 (secure 0 0 0) 0x7f1838099390 con 0x7f1840103ee0 2023-07-14T18:24:14.870 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:14.870 INFO:teuthology.orchestra.run.smithi064.stdout:{"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":376,"monmap":{"epoch":3,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":87,"num_osds":12,"num_up_osds":12,"osd_up_since":1689358943,"num_in_osds":12,"osd_in_since":1689358925,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":102,"data_bytes":656175,"bytes_used":345960448,"bytes_avail":1151594467328,"bytes_total":1151940427776,"recovering_objects_per_sec":0,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":5502,"read_op_per_sec":10,"write_op_per_sec":0},"fsmap":{"epoch":27,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"j","status":"up:standby-replay","gid":14550},{"filesystem_id":1,"rank":3,"name":"d","status":"up:standby-replay","gid":14556},{"filesystem_id":1,"rank":2,"name":"g","status":"up:standby-replay","gid":14616},{"filesystem_id":1,"rank":1,"name":"a","status":"up:standby-replay","gid":14628},{"filesystem_id":1,"rank":0,"name":"e","status":"up:standby-replay","gid":24464},{"filesystem_id":1,"rank":4,"name":"f","status":"up:active","gid":24478},{"filesystem_id":1,"rank":3,"name":"c","status":"up:active","gid":24487},{"filesystem_id":1,"rank":2,"name":"h","status":"up:active","gid":24500},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24508},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24545}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.64:8443/"}},"servicemap":{"epoch":21,"modified":"2023-07-14T18:24:11.475217+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"j":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-07-14T18:24:14.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.819+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 msgr2=0x7f182c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:14.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.819+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 0x7f182c0a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f183c005950 tx=0x7f183c00a300 comp rx=0 tx=0).stop 2023-07-14T18:24:14.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.819+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 msgr2=0x7f1840101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:14.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.819+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f1838007f10 tx=0x7f183800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 shutdown_connections 2023-07-14T18:24:14.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f182c0a11e0 0x7f182c0a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1840103ee0 0x7f1840101310 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f18400fee40 0x7f1840100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 --2- 172.21.15.64:0/2720406419 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f18400ff7e0 0x7f1840100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:14.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 >> 172.21.15.64:0/2720406419 conn(0x7f1840075ca0 msgr2=0x7f1840068240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:14.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 shutdown_connections 2023-07-14T18:24:14.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:14.820+0000 7f1847512700 1 -- 172.21.15.64:0/2720406419 wait complete. 2023-07-14T18:24:15.184 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2023-07-14T18:24:15.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:24:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:15 smithi105 ceph-mon[98013]: pgmap v291: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:15.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:15 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2720406419' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T18:24:15.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:15 smithi064 ceph-mon[96720]: pgmap v291: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:15.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:15 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2720406419' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T18:24:15.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:15 smithi144 ceph-mon[98417]: pgmap v291: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:15.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:15 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2720406419' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T18:24:15.747 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:17.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/3482758851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e580fffb0 msgr2=0x7f7e58100380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:17.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/3482758851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e580fffb0 0x7f7e58100380 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0099c0 tx=0x7f7e4c009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/3482758851 shutdown_connections 2023-07-14T18:24:17.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/3482758851 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e58105000 0x7f7e5810cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/3482758851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e580fffb0 0x7f7e58100380 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/3482758851 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 0x7f7e581048d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/3482758851 >> 172.21.15.64:0/3482758851 conn(0x7f7e58077cd0 msgr2=0x7f7e580780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/3482758851 shutdown_connections 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.103+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/3482758851 wait complete. 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 Processor -- start 2023-07-14T18:24:17.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 -- start start 2023-07-14T18:24:17.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e580fffb0 0x7f7e58072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:17.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 0x7f7e58072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:17.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581a4170 con 0x7f7e58105000 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581a42e0 con 0x7f7e580fffb0 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e5e1ca700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e581a44a0 con 0x7f7e58100950 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e57fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e57fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47936/0 (socket says 172.21.15.64:47936) 2023-07-14T18:24:17.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.104+0000 7f7e57fff700 1 -- 172.21.15.64:0/1391083149 learned_addr learned my addr 172.21.15.64:0/1391083149 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e577fe700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e580fffb0 0x7f7e58072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e56ffd700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 0x7f7e58072e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 -- 172.21.15.64:0/1391083149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 msgr2=0x7f7e58072e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 0x7f7e58072e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 -- 172.21.15.64:0/1391083149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e580fffb0 msgr2=0x7f7e58072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e580fffb0 0x7f7e58072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 -- 172.21.15.64:0/1391083149 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c0095f0 con 0x7f7e58105000 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e57fff700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7e50007d70 tx=0x7f7e5000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5000ad40 con 0x7f7e58105000 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e581a4780 con 0x7f7e58105000 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7e500128b0 con 0x7f7e58105000 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e500115a0 con 0x7f7e58105000 2023-07-14T18:24:17.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.105+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e581a4c80 con 0x7f7e58105000 2023-07-14T18:24:17.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.106+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e5804e340 con 0x7f7e58105000 2023-07-14T18:24:17.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.107+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7e50012a20 con 0x7f7e58105000 2023-07-14T18:24:17.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.108+0000 7f7e54ff9700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 0x7f7e400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:17.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.108+0000 7f7e577fe700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 0x7f7e400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:17.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.108+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7e500cd310 con 0x7f7e58105000 2023-07-14T18:24:17.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.109+0000 7f7e577fe700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 0x7f7e400a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:17.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.112+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7e50096700 con 0x7f7e58105000 2023-07-14T18:24:17.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.309+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7e580fdfc0 con 0x7f7e400a11e0 2023-07-14T18:24:17.376 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.324+0000 7f7e54ff9700 1 -- 172.21.15.64:0/1391083149 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426915 (secure 0 0 0) 0x7f7e580fdfc0 con 0x7f7e400a11e0 2023-07-14T18:24:17.381 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 msgr2=0x7f7e400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:17.383 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 0x7f7e400a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c000c00 tx=0x7f7e4c009f10 comp rx=0 tx=0).stop 2023-07-14T18:24:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 msgr2=0x7f7e58073400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f7e50007d70 tx=0x7f7e5000a340 comp rx=0 tx=0).stop 2023-07-14T18:24:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 shutdown_connections 2023-07-14T18:24:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7e400a11e0 0x7f7e400a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.384 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7e58105000 0x7f7e58073400 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7e580fffb0 0x7f7e58072940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 --2- 172.21.15.64:0/1391083149 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7e58100950 0x7f7e58072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 >> 172.21.15.64:0/1391083149 conn(0x7f7e58077cd0 msgr2=0x7f7e580fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 shutdown_connections 2023-07-14T18:24:17.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:17.327+0000 7f7e5e1ca700 1 -- 172.21.15.64:0/1391083149 wait complete. 2023-07-14T18:24:17.386 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:24:17.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:17 smithi105 ceph-mon[98013]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:17.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:17 smithi064 ceph-mon[96720]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:17.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:17 smithi144 ceph-mon[98417]: pgmap v292: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:17.798 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-07-14T18:24:15.476203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":621,"num_read_kb":430,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":420,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337852,"kb_used_data":13580,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124603972,"statfs":{"total":1151940427776,"available":1151594467328,"internally_reserved":0,"allocated":13905920,"data_stored":7662947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":98,"num_read_kb":48,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.075586"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405474+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:24:07.405474+0000","last_peered":"2023-07-14T18:24:07.405474+0000","last_clean":"2023-07-14T18:24:07.405474+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:24:07.405474+0000","last_undegraded":"2023-07-14T18:24:07.405474+0000","last_fullsized":"2023-07-14T18:24:07.405474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:24:11.348750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405925+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:24:07.405925+0000","last_peered":"2023-07-14T18:24:07.405925+0000","last_clean":"2023-07-14T18:24:07.405925+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:24:07.405925+0000","last_undegraded":"2023-07-14T18:24:07.405925+0000","last_fullsized":"2023-07-14T18:24:07.405925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:08.123484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405720+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:24:07.405720+0000","last_peered":"2023-07-14T18:24:07.405720+0000","last_clean":"2023-07-14T18:24:07.405720+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:24:07.405720+0000","last_undegraded":"2023-07-14T18:24:07.405720+0000","last_fullsized":"2023-07-14T18:24:07.405720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:18.329769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410214+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:24:07.410214+0000","last_peered":"2023-07-14T18:24:07.410214+0000","last_clean":"2023-07-14T18:24:07.410214+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:24:07.410214+0000","last_undegraded":"2023-07-14T18:24:07.410214+0000","last_fullsized":"2023-07-14T18:24:07.410214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:35.873720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403632+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:24:07.403632+0000","last_peered":"2023-07-14T18:24:07.403632+0000","last_clean":"2023-07-14T18:24:07.403632+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:24:07.403632+0000","last_undegraded":"2023-07-14T18:24:07.403632+0000","last_fullsized":"2023-07-14T18:24:07.403632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:47.188696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599111+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:24:07.599111+0000","last_peered":"2023-07-14T18:24:07.599111+0000","last_clean":"2023-07-14T18:24:07.599111+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:24:07.599111+0000","last_undegraded":"2023-07-14T18:24:07.599111+0000","last_fullsized":"2023-07-14T18:24:07.599111+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:17.483400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405673+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:24:07.405673+0000","last_peered":"2023-07-14T18:24:07.405673+0000","last_clean":"2023-07-14T18:24:07.405673+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:24:07.405673+0000","last_undegraded":"2023-07-14T18:24:07.405673+0000","last_fullsized":"2023-07-14T18:24:07.405673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:14.526417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597894+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:24:07.597894+0000","last_peered":"2023-07-14T18:24:07.597894+0000","last_clean":"2023-07-14T18:24:07.597894+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:24:07.597894+0000","last_undegraded":"2023-07-14T18:24:07.597894+0000","last_fullsized":"2023-07-14T18:24:07.597894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:47.842854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413213+0000","last_change":"2023-07-14T18:24:07.413213+0000","last_active":"2023-07-14T18:24:07.413213+0000","last_peered":"2023-07-14T18:24:07.413213+0000","last_clean":"2023-07-14T18:24:07.413213+0000","last_became_active":"2023-07-14T18:24:07.412865+0000","last_became_peered":"2023-07-14T18:24:07.412865+0000","last_unstale":"2023-07-14T18:24:07.413213+0000","last_undegraded":"2023-07-14T18:24:07.413213+0000","last_fullsized":"2023-07-14T18:24:07.413213+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:33.198078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599383+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:24:07.599383+0000","last_peered":"2023-07-14T18:24:07.599383+0000","last_clean":"2023-07-14T18:24:07.599383+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:24:07.599383+0000","last_undegraded":"2023-07-14T18:24:07.599383+0000","last_fullsized":"2023-07-14T18:24:07.599383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:33.854389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406490+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:24:07.406490+0000","last_peered":"2023-07-14T18:24:07.406490+0000","last_clean":"2023-07-14T18:24:07.406490+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:24:07.406490+0000","last_undegraded":"2023-07-14T18:24:07.406490+0000","last_fullsized":"2023-07-14T18:24:07.406490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:44.364403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410203+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:24:07.410203+0000","last_peered":"2023-07-14T18:24:07.410203+0000","last_clean":"2023-07-14T18:24:07.410203+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:24:07.410203+0000","last_undegraded":"2023-07-14T18:24:07.410203+0000","last_fullsized":"2023-07-14T18:24:07.410203+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:33.517728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820341+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:24:07.820341+0000","last_peered":"2023-07-14T18:24:07.820341+0000","last_clean":"2023-07-14T18:24:07.820341+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:24:07.820341+0000","last_undegraded":"2023-07-14T18:24:07.820341+0000","last_fullsized":"2023-07-14T18:24:07.820341+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:47.802767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405516+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:24:07.405516+0000","last_peered":"2023-07-14T18:24:07.405516+0000","last_clean":"2023-07-14T18:24:07.405516+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:24:07.405516+0000","last_undegraded":"2023-07-14T18:24:07.405516+0000","last_fullsized":"2023-07-14T18:24:07.405516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:48:51.690671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405517+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:24:07.405517+0000","last_peered":"2023-07-14T18:24:07.405517+0000","last_clean":"2023-07-14T18:24:07.405517+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:24:07.405517+0000","last_undegraded":"2023-07-14T18:24:07.405517+0000","last_fullsized":"2023-07-14T18:24:07.405517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:59.951987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407820+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:24:07.407820+0000","last_peered":"2023-07-14T18:24:07.407820+0000","last_clean":"2023-07-14T18:24:07.407820+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:24:07.407820+0000","last_undegraded":"2023-07-14T18:24:07.407820+0000","last_fullsized":"2023-07-14T18:24:07.407820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:19.271529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405475+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:24:07.405475+0000","last_peered":"2023-07-14T18:24:07.405475+0000","last_clean":"2023-07-14T18:24:07.405475+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:24:07.405475+0000","last_undegraded":"2023-07-14T18:24:07.405475+0000","last_fullsized":"2023-07-14T18:24:07.405475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:27:22.438148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407722+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:24:07.407722+0000","last_peered":"2023-07-14T18:24:07.407722+0000","last_clean":"2023-07-14T18:24:07.407722+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:24:07.407722+0000","last_undegraded":"2023-07-14T18:24:07.407722+0000","last_fullsized":"2023-07-14T18:24:07.407722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:18.810920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407624+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:24:07.407624+0000","last_peered":"2023-07-14T18:24:07.407624+0000","last_clean":"2023-07-14T18:24:07.407624+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:24:07.407624+0000","last_undegraded":"2023-07-14T18:24:07.407624+0000","last_fullsized":"2023-07-14T18:24:07.407624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:43.490417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406996+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:24:07.406996+0000","last_peered":"2023-07-14T18:24:07.406996+0000","last_clean":"2023-07-14T18:24:07.406996+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:24:07.406996+0000","last_undegraded":"2023-07-14T18:24:07.406996+0000","last_fullsized":"2023-07-14T18:24:07.406996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:27:08.177654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:11.359918+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:24:11.359918+0000","last_peered":"2023-07-14T18:24:11.359918+0000","last_clean":"2023-07-14T18:24:11.359918+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:24:11.359918+0000","last_undegraded":"2023-07-14T18:24:11.359918+0000","last_fullsized":"2023-07-14T18:24:11.359918+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:37.412855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":46,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598331+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:24:07.598331+0000","last_peered":"2023-07-14T18:24:07.598331+0000","last_clean":"2023-07-14T18:24:07.598331+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:24:07.598331+0000","last_undegraded":"2023-07-14T18:24:07.598331+0000","last_fullsized":"2023-07-14T18:24:07.598331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:51.649314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409478+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:24:07.409478+0000","last_peered":"2023-07-14T18:24:07.409478+0000","last_clean":"2023-07-14T18:24:07.409478+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:24:07.409478+0000","last_undegraded":"2023-07-14T18:24:07.409478+0000","last_fullsized":"2023-07-14T18:24:07.409478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:24:08.585476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598686+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:24:07.598686+0000","last_peered":"2023-07-14T18:24:07.598686+0000","last_clean":"2023-07-14T18:24:07.598686+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:24:07.598686+0000","last_undegraded":"2023-07-14T18:24:07.598686+0000","last_fullsized":"2023-07-14T18:24:07.598686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.150320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405997+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:24:07.405997+0000","last_peered":"2023-07-14T18:24:07.405997+0000","last_clean":"2023-07-14T18:24:07.405997+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:24:07.405997+0000","last_undegraded":"2023-07-14T18:24:07.405997+0000","last_fullsized":"2023-07-14T18:24:07.405997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:37:28.657481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409648+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:24:07.409648+0000","last_peered":"2023-07-14T18:24:07.409648+0000","last_clean":"2023-07-14T18:24:07.409648+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:24:07.409648+0000","last_undegraded":"2023-07-14T18:24:07.409648+0000","last_fullsized":"2023-07-14T18:24:07.409648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:20.527639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403960+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:24:07.403960+0000","last_peered":"2023-07-14T18:24:07.403960+0000","last_clean":"2023-07-14T18:24:07.403960+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:24:07.403960+0000","last_undegraded":"2023-07-14T18:24:07.403960+0000","last_fullsized":"2023-07-14T18:24:07.403960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:43.378883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408067+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:24:07.408067+0000","last_peered":"2023-07-14T18:24:07.408067+0000","last_clean":"2023-07-14T18:24:07.408067+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:24:07.408067+0000","last_undegraded":"2023-07-14T18:24:07.408067+0000","last_fullsized":"2023-07-14T18:24:07.408067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:27.452672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409589+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:24:07.409589+0000","last_peered":"2023-07-14T18:24:07.409589+0000","last_clean":"2023-07-14T18:24:07.409589+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:24:07.409589+0000","last_undegraded":"2023-07-14T18:24:07.409589+0000","last_fullsized":"2023-07-14T18:24:07.409589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:26.524872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403362+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:24:07.403362+0000","last_peered":"2023-07-14T18:24:07.403362+0000","last_clean":"2023-07-14T18:24:07.403362+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:24:07.403362+0000","last_undegraded":"2023-07-14T18:24:07.403362+0000","last_fullsized":"2023-07-14T18:24:07.403362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:47.113574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409769+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:24:07.409769+0000","last_peered":"2023-07-14T18:24:07.409769+0000","last_clean":"2023-07-14T18:24:07.409769+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:24:07.409769+0000","last_undegraded":"2023-07-14T18:24:07.409769+0000","last_fullsized":"2023-07-14T18:24:07.409769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:07.113007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.600335+0000","last_change":"2023-07-14T18:24:07.600335+0000","last_active":"2023-07-14T18:24:07.600335+0000","last_peered":"2023-07-14T18:24:07.600335+0000","last_clean":"2023-07-14T18:24:07.600335+0000","last_became_active":"2023-07-14T18:24:07.599841+0000","last_became_peered":"2023-07-14T18:24:07.599841+0000","last_unstale":"2023-07-14T18:24:07.600335+0000","last_undegraded":"2023-07-14T18:24:07.600335+0000","last_fullsized":"2023-07-14T18:24:07.600335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:34.685911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7,11],"acting":[2,10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:11.379183+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:24:11.379183+0000","last_peered":"2023-07-14T18:24:11.379183+0000","last_clean":"2023-07-14T18:24:11.379183+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:24:11.379183+0000","last_undegraded":"2023-07-14T18:24:11.379183+0000","last_fullsized":"2023-07-14T18:24:11.379183+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:31.697182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403600+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:24:07.403600+0000","last_peered":"2023-07-14T18:24:07.403600+0000","last_clean":"2023-07-14T18:24:07.403600+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:24:07.403600+0000","last_undegraded":"2023-07-14T18:24:07.403600+0000","last_fullsized":"2023-07-14T18:24:07.403600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:44.359944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407774+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:24:07.407774+0000","last_peered":"2023-07-14T18:24:07.407774+0000","last_clean":"2023-07-14T18:24:07.407774+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:24:07.407774+0000","last_undegraded":"2023-07-14T18:24:07.407774+0000","last_fullsized":"2023-07-14T18:24:07.407774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:27.518543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599020+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:24:07.599020+0000","last_peered":"2023-07-14T18:24:07.599020+0000","last_clean":"2023-07-14T18:24:07.599020+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:24:07.599020+0000","last_undegraded":"2023-07-14T18:24:07.599020+0000","last_fullsized":"2023-07-14T18:24:07.599020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:57:13.399677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.822326+0000","last_change":"2023-07-14T18:24:07.822326+0000","last_active":"2023-07-14T18:24:07.822326+0000","last_peered":"2023-07-14T18:24:07.822326+0000","last_clean":"2023-07-14T18:24:07.822326+0000","last_became_active":"2023-07-14T18:24:07.821961+0000","last_became_peered":"2023-07-14T18:24:07.821961+0000","last_unstale":"2023-07-14T18:24:07.822326+0000","last_undegraded":"2023-07-14T18:24:07.822326+0000","last_fullsized":"2023-07-14T18:24:07.822326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:49.138102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405293+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:24:07.405293+0000","last_peered":"2023-07-14T18:24:07.405293+0000","last_clean":"2023-07-14T18:24:07.405293+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:24:07.405293+0000","last_undegraded":"2023-07-14T18:24:07.405293+0000","last_fullsized":"2023-07-14T18:24:07.405293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:36.010004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820213+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:24:07.820213+0000","last_peered":"2023-07-14T18:24:07.820213+0000","last_clean":"2023-07-14T18:24:07.820213+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:24:07.820213+0000","last_undegraded":"2023-07-14T18:24:07.820213+0000","last_fullsized":"2023-07-14T18:24:07.820213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:32.445990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598031+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:24:07.598031+0000","last_peered":"2023-07-14T18:24:07.598031+0000","last_clean":"2023-07-14T18:24:07.598031+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:24:07.598031+0000","last_undegraded":"2023-07-14T18:24:07.598031+0000","last_fullsized":"2023-07-14T18:24:07.598031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:38.928448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405774+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:24:07.405774+0000","last_peered":"2023-07-14T18:24:07.405774+0000","last_clean":"2023-07-14T18:24:07.405774+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:24:07.405774+0000","last_undegraded":"2023-07-14T18:24:07.405774+0000","last_fullsized":"2023-07-14T18:24:07.405774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:31.966181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410048+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:24:07.410048+0000","last_peered":"2023-07-14T18:24:07.410048+0000","last_clean":"2023-07-14T18:24:07.410048+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:24:07.410048+0000","last_undegraded":"2023-07-14T18:24:07.410048+0000","last_fullsized":"2023-07-14T18:24:07.410048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:18.846973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407999+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:24:07.407999+0000","last_peered":"2023-07-14T18:24:07.407999+0000","last_clean":"2023-07-14T18:24:07.407999+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:24:07.407999+0000","last_undegraded":"2023-07-14T18:24:07.407999+0000","last_fullsized":"2023-07-14T18:24:07.407999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:36:19.820805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597965+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:24:07.597965+0000","last_peered":"2023-07-14T18:24:07.597965+0000","last_clean":"2023-07-14T18:24:07.597965+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:24:07.597965+0000","last_undegraded":"2023-07-14T18:24:07.597965+0000","last_fullsized":"2023-07-14T18:24:07.597965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:40.703098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":118,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:11.357506+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:24:11.357506+0000","last_peered":"2023-07-14T18:24:11.357506+0000","last_clean":"2023-07-14T18:24:11.357506+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:24:11.357506+0000","last_undegraded":"2023-07-14T18:24:11.357506+0000","last_fullsized":"2023-07-14T18:24:11.357506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:24.616145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408207+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:24:07.408207+0000","last_peered":"2023-07-14T18:24:07.408207+0000","last_clean":"2023-07-14T18:24:07.408207+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:24:07.408207+0000","last_undegraded":"2023-07-14T18:24:07.408207+0000","last_fullsized":"2023-07-14T18:24:07.408207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:44:26.421835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413138+0000","last_change":"2023-07-14T18:24:07.412404+0000","last_active":"2023-07-14T18:24:07.413138+0000","last_peered":"2023-07-14T18:24:07.413138+0000","last_clean":"2023-07-14T18:24:07.413138+0000","last_became_active":"2023-07-14T18:24:07.411933+0000","last_became_peered":"2023-07-14T18:24:07.411933+0000","last_unstale":"2023-07-14T18:24:07.413138+0000","last_undegraded":"2023-07-14T18:24:07.413138+0000","last_fullsized":"2023-07-14T18:24:07.413138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:22.236815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406710+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:24:07.406710+0000","last_peered":"2023-07-14T18:24:07.406710+0000","last_clean":"2023-07-14T18:24:07.406710+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:24:07.406710+0000","last_undegraded":"2023-07-14T18:24:07.406710+0000","last_fullsized":"2023-07-14T18:24:07.406710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:32:56.571997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.411973+0000","last_change":"2023-07-14T18:24:07.411973+0000","last_active":"2023-07-14T18:24:07.411973+0000","last_peered":"2023-07-14T18:24:07.411973+0000","last_clean":"2023-07-14T18:24:07.411973+0000","last_became_active":"2023-07-14T18:24:07.411638+0000","last_became_peered":"2023-07-14T18:24:07.411638+0000","last_unstale":"2023-07-14T18:24:07.411973+0000","last_undegraded":"2023-07-14T18:24:07.411973+0000","last_fullsized":"2023-07-14T18:24:07.411973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:28.531130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413964+0000","last_change":"2023-07-14T18:24:07.413964+0000","last_active":"2023-07-14T18:24:07.413964+0000","last_peered":"2023-07-14T18:24:07.413964+0000","last_clean":"2023-07-14T18:24:07.413964+0000","last_became_active":"2023-07-14T18:24:07.413507+0000","last_became_peered":"2023-07-14T18:24:07.413507+0000","last_unstale":"2023-07-14T18:24:07.413964+0000","last_undegraded":"2023-07-14T18:24:07.413964+0000","last_fullsized":"2023-07-14T18:24:07.413964+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:27.116037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409637+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:24:07.409637+0000","last_peered":"2023-07-14T18:24:07.409637+0000","last_clean":"2023-07-14T18:24:07.409637+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:24:07.409637+0000","last_undegraded":"2023-07-14T18:24:07.409637+0000","last_fullsized":"2023-07-14T18:24:07.409637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:25.197615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408340+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:24:07.408340+0000","last_peered":"2023-07-14T18:24:07.408340+0000","last_clean":"2023-07-14T18:24:07.408340+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:24:07.408340+0000","last_undegraded":"2023-07-14T18:24:07.408340+0000","last_fullsized":"2023-07-14T18:24:07.408340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:15:33.259444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407780+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:24:07.407780+0000","last_peered":"2023-07-14T18:24:07.407780+0000","last_clean":"2023-07-14T18:24:07.407780+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:24:07.407780+0000","last_undegraded":"2023-07-14T18:24:07.407780+0000","last_fullsized":"2023-07-14T18:24:07.407780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:42.309537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'33","reported_seq":153,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:11.357399+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:24:11.357399+0000","last_peered":"2023-07-14T18:24:11.357399+0000","last_clean":"2023-07-14T18:24:11.357399+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:24:11.357399+0000","last_undegraded":"2023-07-14T18:24:11.357399+0000","last_fullsized":"2023-07-14T18:24:11.357399+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:00.246915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":94,"num_read_kb":103,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403773+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:24:07.403773+0000","last_peered":"2023-07-14T18:24:07.403773+0000","last_clean":"2023-07-14T18:24:07.403773+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:24:07.403773+0000","last_undegraded":"2023-07-14T18:24:07.403773+0000","last_fullsized":"2023-07-14T18:24:07.403773+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:19.677064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410030+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:24:07.410030+0000","last_peered":"2023-07-14T18:24:07.410030+0000","last_clean":"2023-07-14T18:24:07.410030+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:24:07.410030+0000","last_undegraded":"2023-07-14T18:24:07.410030+0000","last_fullsized":"2023-07-14T18:24:07.410030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:15.831940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407198+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:24:07.407198+0000","last_peered":"2023-07-14T18:24:07.407198+0000","last_clean":"2023-07-14T18:24:07.407198+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:24:07.407198+0000","last_undegraded":"2023-07-14T18:24:07.407198+0000","last_fullsized":"2023-07-14T18:24:07.407198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:03.898501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820320+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:24:07.820320+0000","last_peered":"2023-07-14T18:24:07.820320+0000","last_clean":"2023-07-14T18:24:07.820320+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:24:07.820320+0000","last_undegraded":"2023-07-14T18:24:07.820320+0000","last_fullsized":"2023-07-14T18:24:07.820320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:28:38.028092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405119+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:24:07.405119+0000","last_peered":"2023-07-14T18:24:07.405119+0000","last_clean":"2023-07-14T18:24:07.405119+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:24:07.405119+0000","last_undegraded":"2023-07-14T18:24:07.405119+0000","last_fullsized":"2023-07-14T18:24:07.405119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:57.627580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409613+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:24:07.409613+0000","last_peered":"2023-07-14T18:24:07.409613+0000","last_clean":"2023-07-14T18:24:07.409613+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:24:07.409613+0000","last_undegraded":"2023-07-14T18:24:07.409613+0000","last_fullsized":"2023-07-14T18:24:07.409613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:13.349187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405811+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:24:07.405811+0000","last_peered":"2023-07-14T18:24:07.405811+0000","last_clean":"2023-07-14T18:24:07.405811+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:24:07.405811+0000","last_undegraded":"2023-07-14T18:24:07.405811+0000","last_fullsized":"2023-07-14T18:24:07.405811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:17:48.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407367+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:24:07.407367+0000","last_peered":"2023-07-14T18:24:07.407367+0000","last_clean":"2023-07-14T18:24:07.407367+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:24:07.407367+0000","last_undegraded":"2023-07-14T18:24:07.407367+0000","last_fullsized":"2023-07-14T18:24:07.407367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:31:02.627246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:13.363395+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:24:13.363395+0000","last_peered":"2023-07-14T18:24:13.363395+0000","last_clean":"2023-07-14T18:24:13.363395+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:24:13.363395+0000","last_undegraded":"2023-07-14T18:24:13.363395+0000","last_fullsized":"2023-07-14T18:24:13.363395+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:05.604630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405854+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:24:07.405854+0000","last_peered":"2023-07-14T18:24:07.405854+0000","last_clean":"2023-07-14T18:24:07.405854+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:24:07.405854+0000","last_undegraded":"2023-07-14T18:24:07.405854+0000","last_fullsized":"2023-07-14T18:24:07.405854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.422448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820996+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:24:07.820996+0000","last_peered":"2023-07-14T18:24:07.820996+0000","last_clean":"2023-07-14T18:24:07.820996+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:24:07.820996+0000","last_undegraded":"2023-07-14T18:24:07.820996+0000","last_fullsized":"2023-07-14T18:24:07.820996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:00.885274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403130+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:24:07.403130+0000","last_peered":"2023-07-14T18:24:07.403130+0000","last_clean":"2023-07-14T18:24:07.403130+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:24:07.403130+0000","last_undegraded":"2023-07-14T18:24:07.403130+0000","last_fullsized":"2023-07-14T18:24:07.403130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:22.401795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413544+0000","last_change":"2023-07-14T18:24:07.413544+0000","last_active":"2023-07-14T18:24:07.413544+0000","last_peered":"2023-07-14T18:24:07.413544+0000","last_clean":"2023-07-14T18:24:07.413544+0000","last_became_active":"2023-07-14T18:24:07.413217+0000","last_became_peered":"2023-07-14T18:24:07.413217+0000","last_unstale":"2023-07-14T18:24:07.413544+0000","last_undegraded":"2023-07-14T18:24:07.413544+0000","last_fullsized":"2023-07-14T18:24:07.413544+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:24.792524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410040+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:24:07.410040+0000","last_peered":"2023-07-14T18:24:07.410040+0000","last_clean":"2023-07-14T18:24:07.410040+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:24:07.410040+0000","last_undegraded":"2023-07-14T18:24:07.410040+0000","last_fullsized":"2023-07-14T18:24:07.410040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:18.284526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404072+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:24:07.404072+0000","last_peered":"2023-07-14T18:24:07.404072+0000","last_clean":"2023-07-14T18:24:07.404072+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:24:07.404072+0000","last_undegraded":"2023-07-14T18:24:07.404072+0000","last_fullsized":"2023-07-14T18:24:07.404072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:44.906119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405246+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:24:07.405246+0000","last_peered":"2023-07-14T18:24:07.405246+0000","last_clean":"2023-07-14T18:24:07.405246+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:24:07.405246+0000","last_undegraded":"2023-07-14T18:24:07.405246+0000","last_fullsized":"2023-07-14T18:24:07.405246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:41.007811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403367+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:24:07.403367+0000","last_peered":"2023-07-14T18:24:07.403367+0000","last_clean":"2023-07-14T18:24:07.403367+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:24:07.403367+0000","last_undegraded":"2023-07-14T18:24:07.403367+0000","last_fullsized":"2023-07-14T18:24:07.403367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:06.743747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413261+0000","last_change":"2023-07-14T18:24:07.413261+0000","last_active":"2023-07-14T18:24:07.413261+0000","last_peered":"2023-07-14T18:24:07.413261+0000","last_clean":"2023-07-14T18:24:07.413261+0000","last_became_active":"2023-07-14T18:24:07.412759+0000","last_became_peered":"2023-07-14T18:24:07.412759+0000","last_unstale":"2023-07-14T18:24:07.413261+0000","last_undegraded":"2023-07-14T18:24:07.413261+0000","last_fullsized":"2023-07-14T18:24:07.413261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:09.725334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599408+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:24:07.599408+0000","last_peered":"2023-07-14T18:24:07.599408+0000","last_clean":"2023-07-14T18:24:07.599408+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:24:07.599408+0000","last_undegraded":"2023-07-14T18:24:07.599408+0000","last_fullsized":"2023-07-14T18:24:07.599408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:28:33.092745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820487+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:24:07.820487+0000","last_peered":"2023-07-14T18:24:07.820487+0000","last_clean":"2023-07-14T18:24:07.820487+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:24:07.820487+0000","last_undegraded":"2023-07-14T18:24:07.820487+0000","last_fullsized":"2023-07-14T18:24:07.820487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:09:47.913784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:12.361970+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:24:12.361970+0000","last_peered":"2023-07-14T18:24:12.361970+0000","last_clean":"2023-07-14T18:24:12.361970+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:24:12.361970+0000","last_undegraded":"2023-07-14T18:24:12.361970+0000","last_fullsized":"2023-07-14T18:24:12.361970+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:23.957543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":47,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599127+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:24:07.599127+0000","last_peered":"2023-07-14T18:24:07.599127+0000","last_clean":"2023-07-14T18:24:07.599127+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:24:07.599127+0000","last_undegraded":"2023-07-14T18:24:07.599127+0000","last_fullsized":"2023-07-14T18:24:07.599127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:22.907702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821088+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:24:07.821088+0000","last_peered":"2023-07-14T18:24:07.821088+0000","last_clean":"2023-07-14T18:24:07.821088+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:24:07.821088+0000","last_undegraded":"2023-07-14T18:24:07.821088+0000","last_fullsized":"2023-07-14T18:24:07.821088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:26.669137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410319+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:24:07.410319+0000","last_peered":"2023-07-14T18:24:07.410319+0000","last_clean":"2023-07-14T18:24:07.410319+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:24:07.410319+0000","last_undegraded":"2023-07-14T18:24:07.410319+0000","last_fullsized":"2023-07-14T18:24:07.410319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:11.487403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:13.362607+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:24:13.362607+0000","last_peered":"2023-07-14T18:24:13.362607+0000","last_clean":"2023-07-14T18:24:13.362607+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:24:13.362607+0000","last_undegraded":"2023-07-14T18:24:13.362607+0000","last_fullsized":"2023-07-14T18:24:13.362607+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:44.654417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406370+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:24:07.406370+0000","last_peered":"2023-07-14T18:24:07.406370+0000","last_clean":"2023-07-14T18:24:07.406370+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:24:07.406370+0000","last_undegraded":"2023-07-14T18:24:07.406370+0000","last_fullsized":"2023-07-14T18:24:07.406370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:01.561467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:01.628715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409780+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:24:07.409780+0000","last_peered":"2023-07-14T18:24:07.409780+0000","last_clean":"2023-07-14T18:24:07.409780+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:24:07.409780+0000","last_undegraded":"2023-07-14T18:24:07.409780+0000","last_fullsized":"2023-07-14T18:24:07.409780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:45:44.875754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405706+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:24:07.405706+0000","last_peered":"2023-07-14T18:24:07.405706+0000","last_clean":"2023-07-14T18:24:07.405706+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:24:07.405706+0000","last_undegraded":"2023-07-14T18:24:07.405706+0000","last_fullsized":"2023-07-14T18:24:07.405706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:07.000585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410311+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:24:07.410311+0000","last_peered":"2023-07-14T18:24:07.410311+0000","last_clean":"2023-07-14T18:24:07.410311+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:24:07.410311+0000","last_undegraded":"2023-07-14T18:24:07.410311+0000","last_fullsized":"2023-07-14T18:24:07.410311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:22.806069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820058+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:24:07.820058+0000","last_peered":"2023-07-14T18:24:07.820058+0000","last_clean":"2023-07-14T18:24:07.820058+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:24:07.820058+0000","last_undegraded":"2023-07-14T18:24:07.820058+0000","last_fullsized":"2023-07-14T18:24:07.820058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:00.275183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405700+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:24:07.405700+0000","last_peered":"2023-07-14T18:24:07.405700+0000","last_clean":"2023-07-14T18:24:07.405700+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:24:07.405700+0000","last_undegraded":"2023-07-14T18:24:07.405700+0000","last_fullsized":"2023-07-14T18:24:07.405700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:04.719205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405561+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:24:07.405561+0000","last_peered":"2023-07-14T18:24:07.405561+0000","last_clean":"2023-07-14T18:24:07.405561+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:24:07.405561+0000","last_undegraded":"2023-07-14T18:24:07.405561+0000","last_fullsized":"2023-07-14T18:24:07.405561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:44.840479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403787+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:24:07.403787+0000","last_peered":"2023-07-14T18:24:07.403787+0000","last_clean":"2023-07-14T18:24:07.403787+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:24:07.403787+0000","last_undegraded":"2023-07-14T18:24:07.403787+0000","last_fullsized":"2023-07-14T18:24:07.403787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.485409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820701+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:24:07.820701+0000","last_peered":"2023-07-14T18:24:07.820701+0000","last_clean":"2023-07-14T18:24:07.820701+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:24:07.820701+0000","last_undegraded":"2023-07-14T18:24:07.820701+0000","last_fullsized":"2023-07-14T18:24:07.820701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:52:09.481241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597956+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:24:07.597956+0000","last_peered":"2023-07-14T18:24:07.597956+0000","last_clean":"2023-07-14T18:24:07.597956+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:24:07.597956+0000","last_undegraded":"2023-07-14T18:24:07.597956+0000","last_fullsized":"2023-07-14T18:24:07.597956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:55.573942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407882+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:24:07.407882+0000","last_peered":"2023-07-14T18:24:07.407882+0000","last_clean":"2023-07-14T18:24:07.407882+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:24:07.407882+0000","last_undegraded":"2023-07-14T18:24:07.407882+0000","last_fullsized":"2023-07-14T18:24:07.407882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:55:27.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598511+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:24:07.598511+0000","last_peered":"2023-07-14T18:24:07.598511+0000","last_clean":"2023-07-14T18:24:07.598511+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:24:07.598511+0000","last_undegraded":"2023-07-14T18:24:07.598511+0000","last_fullsized":"2023-07-14T18:24:07.598511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:57.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406745+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:24:07.406745+0000","last_peered":"2023-07-14T18:24:07.406745+0000","last_clean":"2023-07-14T18:24:07.406745+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:24:07.406745+0000","last_undegraded":"2023-07-14T18:24:07.406745+0000","last_fullsized":"2023-07-14T18:24:07.406745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:40.891887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410518+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:24:07.410518+0000","last_peered":"2023-07-14T18:24:07.410518+0000","last_clean":"2023-07-14T18:24:07.410518+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:24:07.410518+0000","last_undegraded":"2023-07-14T18:24:07.410518+0000","last_fullsized":"2023-07-14T18:24:07.410518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:54.251569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405928+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:24:07.405928+0000","last_peered":"2023-07-14T18:24:07.405928+0000","last_clean":"2023-07-14T18:24:07.405928+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:24:07.405928+0000","last_undegraded":"2023-07-14T18:24:07.405928+0000","last_fullsized":"2023-07-14T18:24:07.405928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:46.633579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599735+0000","last_change":"2023-07-14T18:24:07.599735+0000","last_active":"2023-07-14T18:24:07.599735+0000","last_peered":"2023-07-14T18:24:07.599735+0000","last_clean":"2023-07-14T18:24:07.599735+0000","last_became_active":"2023-07-14T18:24:07.599353+0000","last_became_peered":"2023-07-14T18:24:07.599353+0000","last_unstale":"2023-07-14T18:24:07.599735+0000","last_undegraded":"2023-07-14T18:24:07.599735+0000","last_fullsized":"2023-07-14T18:24:07.599735+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:36.613425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407843+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:24:07.407843+0000","last_peered":"2023-07-14T18:24:07.407843+0000","last_clean":"2023-07-14T18:24:07.407843+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:24:07.407843+0000","last_undegraded":"2023-07-14T18:24:07.407843+0000","last_fullsized":"2023-07-14T18:24:07.407843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:27:45.050723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403163+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:24:07.403163+0000","last_peered":"2023-07-14T18:24:07.403163+0000","last_clean":"2023-07-14T18:24:07.403163+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:24:07.403163+0000","last_undegraded":"2023-07-14T18:24:07.403163+0000","last_fullsized":"2023-07-14T18:24:07.403163+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:15:37.710301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407589+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:24:07.407589+0000","last_peered":"2023-07-14T18:24:07.407589+0000","last_clean":"2023-07-14T18:24:07.407589+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:24:07.407589+0000","last_undegraded":"2023-07-14T18:24:07.407589+0000","last_fullsized":"2023-07-14T18:24:07.407589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:02.479091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:37.977157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404952+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:24:07.404952+0000","last_peered":"2023-07-14T18:24:07.404952+0000","last_clean":"2023-07-14T18:24:07.404952+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:24:07.404952+0000","last_undegraded":"2023-07-14T18:24:07.404952+0000","last_fullsized":"2023-07-14T18:24:07.404952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:37.080797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410069+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:24:07.410069+0000","last_peered":"2023-07-14T18:24:07.410069+0000","last_clean":"2023-07-14T18:24:07.410069+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:24:07.410069+0000","last_undegraded":"2023-07-14T18:24:07.410069+0000","last_fullsized":"2023-07-14T18:24:07.410069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:00.967275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407086+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:24:07.407086+0000","last_peered":"2023-07-14T18:24:07.407086+0000","last_clean":"2023-07-14T18:24:07.407086+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:24:07.407086+0000","last_undegraded":"2023-07-14T18:24:07.407086+0000","last_fullsized":"2023-07-14T18:24:07.407086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:39.830952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598357+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:24:07.598357+0000","last_peered":"2023-07-14T18:24:07.598357+0000","last_clean":"2023-07-14T18:24:07.598357+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:24:07.598357+0000","last_undegraded":"2023-07-14T18:24:07.598357+0000","last_fullsized":"2023-07-14T18:24:07.598357+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:12.153021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409937+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:24:07.409937+0000","last_peered":"2023-07-14T18:24:07.409937+0000","last_clean":"2023-07-14T18:24:07.409937+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:24:07.409937+0000","last_undegraded":"2023-07-14T18:24:07.409937+0000","last_fullsized":"2023-07-14T18:24:07.409937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:41.469044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.416915+0000","last_change":"2023-07-14T18:24:07.416556+0000","last_active":"2023-07-14T18:24:07.416915+0000","last_peered":"2023-07-14T18:24:07.416915+0000","last_clean":"2023-07-14T18:24:07.416915+0000","last_became_active":"2023-07-14T18:24:07.408596+0000","last_became_peered":"2023-07-14T18:24:07.408596+0000","last_unstale":"2023-07-14T18:24:07.416915+0000","last_undegraded":"2023-07-14T18:24:07.416915+0000","last_fullsized":"2023-07-14T18:24:07.416915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:10.303257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405892+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:24:07.405892+0000","last_peered":"2023-07-14T18:24:07.405892+0000","last_clean":"2023-07-14T18:24:07.405892+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:24:07.405892+0000","last_undegraded":"2023-07-14T18:24:07.405892+0000","last_fullsized":"2023-07-14T18:24:07.405892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:59:09.124887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407944+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:24:07.407944+0000","last_peered":"2023-07-14T18:24:07.407944+0000","last_clean":"2023-07-14T18:24:07.407944+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:24:07.407944+0000","last_undegraded":"2023-07-14T18:24:07.407944+0000","last_fullsized":"2023-07-14T18:24:07.407944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:33.118185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:11.357931+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:24:11.357931+0000","last_peered":"2023-07-14T18:24:11.357931+0000","last_clean":"2023-07-14T18:24:11.357931+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:24:11.357931+0000","last_undegraded":"2023-07-14T18:24:11.357931+0000","last_fullsized":"2023-07-14T18:24:11.357931+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:33.188924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406610+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:24:07.406610+0000","last_peered":"2023-07-14T18:24:07.406610+0000","last_clean":"2023-07-14T18:24:07.406610+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:24:07.406610+0000","last_undegraded":"2023-07-14T18:24:07.406610+0000","last_fullsized":"2023-07-14T18:24:07.406610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:25:28.457835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406508+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:24:07.406508+0000","last_peered":"2023-07-14T18:24:07.406508+0000","last_clean":"2023-07-14T18:24:07.406508+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:24:07.406508+0000","last_undegraded":"2023-07-14T18:24:07.406508+0000","last_fullsized":"2023-07-14T18:24:07.406508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:42:15.619889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409875+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:24:07.409875+0000","last_peered":"2023-07-14T18:24:07.409875+0000","last_clean":"2023-07-14T18:24:07.409875+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:24:07.409875+0000","last_undegraded":"2023-07-14T18:24:07.409875+0000","last_fullsized":"2023-07-14T18:24:07.409875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:21:05.734566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406230+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:24:07.406230+0000","last_peered":"2023-07-14T18:24:07.406230+0000","last_clean":"2023-07-14T18:24:07.406230+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:24:07.406230+0000","last_undegraded":"2023-07-14T18:24:07.406230+0000","last_fullsized":"2023-07-14T18:24:07.406230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:18.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408921+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:24:07.408921+0000","last_peered":"2023-07-14T18:24:07.408921+0000","last_clean":"2023-07-14T18:24:07.408921+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:24:07.408921+0000","last_undegraded":"2023-07-14T18:24:07.408921+0000","last_fullsized":"2023-07-14T18:24:07.408921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:45.492125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821139+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:24:07.821139+0000","last_peered":"2023-07-14T18:24:07.821139+0000","last_clean":"2023-07-14T18:24:07.821139+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:24:07.821139+0000","last_undegraded":"2023-07-14T18:24:07.821139+0000","last_fullsized":"2023-07-14T18:24:07.821139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:41.844783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407181+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:24:07.407181+0000","last_peered":"2023-07-14T18:24:07.407181+0000","last_clean":"2023-07-14T18:24:07.407181+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:24:07.407181+0000","last_undegraded":"2023-07-14T18:24:07.407181+0000","last_fullsized":"2023-07-14T18:24:07.407181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:34:35.248238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405025+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:24:07.405025+0000","last_peered":"2023-07-14T18:24:07.405025+0000","last_clean":"2023-07-14T18:24:07.405025+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:24:07.405025+0000","last_undegraded":"2023-07-14T18:24:07.405025+0000","last_fullsized":"2023-07-14T18:24:07.405025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:53.390475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406959+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:24:07.406959+0000","last_peered":"2023-07-14T18:24:07.406959+0000","last_clean":"2023-07-14T18:24:07.406959+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:24:07.406959+0000","last_undegraded":"2023-07-14T18:24:07.406959+0000","last_fullsized":"2023-07-14T18:24:07.406959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:26.096564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406915+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:24:07.406915+0000","last_peered":"2023-07-14T18:24:07.406915+0000","last_clean":"2023-07-14T18:24:07.406915+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:24:07.406915+0000","last_undegraded":"2023-07-14T18:24:07.406915+0000","last_fullsized":"2023-07-14T18:24:07.406915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:32:24.873296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406046+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:24:07.406046+0000","last_peered":"2023-07-14T18:24:07.406046+0000","last_clean":"2023-07-14T18:24:07.406046+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:24:07.406046+0000","last_undegraded":"2023-07-14T18:24:07.406046+0000","last_fullsized":"2023-07-14T18:24:07.406046+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:58.884740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820615+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:24:07.820615+0000","last_peered":"2023-07-14T18:24:07.820615+0000","last_clean":"2023-07-14T18:24:07.820615+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:24:07.820615+0000","last_undegraded":"2023-07-14T18:24:07.820615+0000","last_fullsized":"2023-07-14T18:24:07.820615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:23:26.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405289+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:24:07.405289+0000","last_peered":"2023-07-14T18:24:07.405289+0000","last_clean":"2023-07-14T18:24:07.405289+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:24:07.405289+0000","last_undegraded":"2023-07-14T18:24:07.405289+0000","last_fullsized":"2023-07-14T18:24:07.405289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:02.717582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407277+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:24:07.407277+0000","last_peered":"2023-07-14T18:24:07.407277+0000","last_clean":"2023-07-14T18:24:07.407277+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:24:07.407277+0000","last_undegraded":"2023-07-14T18:24:07.407277+0000","last_fullsized":"2023-07-14T18:24:07.407277+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:15:50.630547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598631+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:24:07.598631+0000","last_peered":"2023-07-14T18:24:07.598631+0000","last_clean":"2023-07-14T18:24:07.598631+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:24:07.598631+0000","last_undegraded":"2023-07-14T18:24:07.598631+0000","last_fullsized":"2023-07-14T18:24:07.598631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:06.982685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410189+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:24:07.410189+0000","last_peered":"2023-07-14T18:24:07.410189+0000","last_clean":"2023-07-14T18:24:07.410189+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:24:07.410189+0000","last_undegraded":"2023-07-14T18:24:07.410189+0000","last_fullsized":"2023-07-14T18:24:07.410189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:49:31.338811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598143+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:24:07.598143+0000","last_peered":"2023-07-14T18:24:07.598143+0000","last_clean":"2023-07-14T18:24:07.598143+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:24:07.598143+0000","last_undegraded":"2023-07-14T18:24:07.598143+0000","last_fullsized":"2023-07-14T18:24:07.598143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408224+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:24:07.408224+0000","last_peered":"2023-07-14T18:24:07.408224+0000","last_clean":"2023-07-14T18:24:07.408224+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:24:07.408224+0000","last_undegraded":"2023-07-14T18:24:07.408224+0000","last_fullsized":"2023-07-14T18:24:07.408224+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.457179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820178+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:24:07.820178+0000","last_peered":"2023-07-14T18:24:07.820178+0000","last_clean":"2023-07-14T18:24:07.820178+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:24:07.820178+0000","last_undegraded":"2023-07-14T18:24:07.820178+0000","last_fullsized":"2023-07-14T18:24:07.820178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:24.150124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405223+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:24:07.405223+0000","last_peered":"2023-07-14T18:24:07.405223+0000","last_clean":"2023-07-14T18:24:07.405223+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:24:07.405223+0000","last_undegraded":"2023-07-14T18:24:07.405223+0000","last_fullsized":"2023-07-14T18:24:07.405223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:22.497297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":485,"num_read_kb":311,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514521,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":477638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.48199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.66300000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.624}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.69899999999999995}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.754}]},{"osd":8,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.77000000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.83399999999999996}]}]},{"osd":10,"up_from":69,"seq":296352743453,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1122615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.436}]},{"osd":1,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63,"15min":0.63},"min":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.78600000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47999999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53700000000000003}]},{"osd":4,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.373}]},{"osd":5,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.58699999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.82999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.872,"5min":1.052,"15min":1.052},"last":0.56399999999999995}]},{"osd":6,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.58099999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.58699999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.68300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.92600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.80200000000000005},"min":{"1min":0.64600000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.623}]},{"osd":9,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.57599999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.87,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84699999999999998,"5min":1.256,"15min":1.256},"last":0.64900000000000002}]},{"osd":11,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.57699999999999996}]}]},{"osd":9,"up_from":64,"seq":274877906978,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":481322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.499,"15min":0.46400000000000002},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":5.8559999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.874,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.5140000000000002,"5min":5.5140000000000002,"15min":5.5140000000000002},"last":0.64300000000000002}]},{"osd":2,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.441},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.71299999999999997,"15min":0.68799999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.641,"5min":5.641,"15min":5.641},"last":0.51300000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.377},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.39600000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5679999999999996,"5min":5.5679999999999996,"15min":5.5679999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.57999999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78200000000000003,"15min":0.79400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.55700000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80400000000000005,"15min":0.81399999999999995},"min":{"1min":0.53000000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.63200000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.85299999999999998,"15min":0.82899999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.875},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.70899999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.70199999999999996}]},{"osd":11,"last update":"Fri Jul 14 18:24:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.61299999999999999}]}]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":477604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.443}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39000000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.1280000000000001,"5min":3.1280000000000001,"15min":3.1280000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.625}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.58399999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.59599999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.48299999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.64800000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70399999999999996,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.55200000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.71899999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.872,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.68000000000000005}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.69699999999999995}]}]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1073683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.45000000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.46400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.60499999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.64700000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.38200000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.48099999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89600000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.86199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.79000000000000004}]},{"osd":6,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92500000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90500000000000003,"5min":1.02,"15min":1.02},"last":0.626}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.83099999999999996}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97099999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.752}]},{"osd":10,"last update":"Fri Jul 14 18:24:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86199999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.66300000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.876,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.78100000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86499999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":472409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.442},{"interface":"front","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40500000000000003},"min":{"1min":0.28299999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.42299999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.60499999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.51300000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93300000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.45800000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96399999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.70899999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.80100000000000005,"5min":1.099,"15min":1.099},"last":0.72199999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.64100000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.627},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.65300000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76000000000000001,"15min":0.76500000000000001},"min":{"1min":0.63,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.752,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Fri Jul 14 18:24:11 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.76000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":532263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.42599999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.441},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.47199999999999998}]},{"osd":3,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.53400000000000003},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.42999999999999999},"min":{"1min":0.35799999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71199999999999997,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.48499999999999999,"15min":0.46200000000000002},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.76700000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.35199999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.497}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.64000000000000001,"15min":0.61299999999999999},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93999999999999995,"5min":1.089,"15min":1.089},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.64000000000000001,"15min":0.61499999999999999},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63900000000000001}]},{"osd":7,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.498,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0249999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.623,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88900000000000001,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.55400000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69499999999999995,"15min":0.67900000000000005},"min":{"1min":0.54600000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.012,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.53100000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":1.173,"15min":1.173},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.097,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.63},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.65900000000000003}]},{"osd":11,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.62},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":0.64800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":474255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56899999999999995,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.66200000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81100000000000005,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.96599999999999997,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.624}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.69099999999999995}]},{"osd":4,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.54700000000000004},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.61499999999999999,"15min":0.57099999999999995},"min":{"1min":0.498,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.84399999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.48599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89800000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.59899999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.71699999999999997}]},{"osd":7,"last update":"Fri Jul 14 18:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.91300000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97399999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.752}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.78500000000000003},"min":{"1min":0.503,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.85499999999999998,"5min":1.036,"15min":1.036},"last":0.88300000000000001}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.442,"15min":0.442},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.65500000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91400000000000003,"5min":1.075,"15min":1.075},"last":0.97599999999999998}]},{"osd":11,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.85999999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":533215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.53700000000000003}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.46000000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60699999999999998}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.374},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.39400000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.54100000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.497},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.434}]},{"osd":7,"last update":"Fri Jul 14 18:22:59 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.88,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.66400000000000003,"15min":0.66100000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.88900000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.68100000000000005}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.45200000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.72299999999999998,"15min":0.70499999999999996},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.58399999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.621,"15min":1.621},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.627}]},{"osd":11,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.54700000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":472157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41199999999999998,"15min":0.374},"min":{"1min":0.29099999999999998,"5min":0.161,"15min":0.161},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.497},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.47799999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"last":0.40300000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.629,"15min":0.57899999999999996},"min":{"1min":0.41799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":8.1820000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.68200000000000005,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.2210000000000001,"5min":8.2210000000000001,"15min":8.2210000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.63800000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.68500000000000005,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.69199999999999995,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.77800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.81000000000000005,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":11.609999999999999,"5min":11.609999999999999,"15min":11.609999999999999},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.125,"5min":0.84499999999999997,"15min":0.79900000000000004},"min":{"1min":0.497,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.0559999999999992,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.28,"5min":0.879,"15min":0.81200000000000006},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.90400000000000003,"15min":0.83799999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":12.291,"5min":12.291,"15min":12.291},"last":0.68200000000000005}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":11.263999999999999,"5min":11.263999999999999,"15min":11.263999999999999},"last":0.754},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.1120000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":476483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.37},"min":{"1min":0.29999999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54400000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.441},{"interface":"front","average":{"1min":0.437,"5min":0.41899999999999998,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.621,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.47499999999999998}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.57499999999999996,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.56200000000000006},"min":{"1min":0.495,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56100000000000005,"15min":0.53000000000000003},"min":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78300000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.627},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.52800000000000002}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89000000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.78900000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.60499999999999998},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.44700000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72499999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.875},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.80300000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.83499999999999996}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.74199999999999999,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.98099999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.76600000000000001}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.90900000000000003}]}]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1601536,"data_stored":1069303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.46200000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.628}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.438}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.52900000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83399999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.66400000000000003}]},{"osd":7,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68000000000000005,"15min":0.66000000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94499999999999995,"5min":1.075,"15min":1.075},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84699999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.61499999999999999}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89600000000000002,"5min":1.087,"15min":1.087},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.80200000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.98799999999999999,"5min":1.206,"15min":1.206},"last":0.77000000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95699999999999996,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.747},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77100000000000002},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.873,"5min":1.175,"15min":1.175},"last":0.68200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:24:18.561 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:18 smithi105 ceph-mon[98013]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:18.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:18 smithi064 ceph-mon[96720]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:18.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:18 smithi144 ceph-mon[98417]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:19.562 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:19 smithi105 ceph-mon[98013]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:19.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:19 smithi064 ceph-mon[96720]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:19.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:19 smithi144 ceph-mon[98417]: pgmap v293: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:20.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T18:24:21.385 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:21.570 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:21 smithi064 ceph-mon[96720]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:21.682 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:21 smithi144 ceph-mon[98417]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:21.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:21 smithi105 ceph-mon[98013]: pgmap v294: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 10 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T18:24:22.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.534+0000 7f18416f3700 1 -- 172.21.15.64:0/3979234689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 msgr2=0x7f183c0ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:22.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.534+0000 7f18416f3700 1 --2- 172.21.15.64:0/3979234689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c0ff260 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f18300099c0 tx=0x7f1830009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 -- 172.21.15.64:0/3979234689 shutdown_connections 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 --2- 172.21.15.64:0/3979234689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c103f30 0x7f183c10bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 --2- 172.21.15.64:0/3979234689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c0ff260 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 --2- 172.21.15.64:0/3979234689 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c0ff830 0x7f183c103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 -- 172.21.15.64:0/3979234689 >> 172.21.15.64:0/3979234689 conn(0x7f183c075ae0 msgr2=0x7f183c075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 -- 172.21.15.64:0/3979234689 shutdown_connections 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 -- 172.21.15.64:0/3979234689 wait complete. 2023-07-14T18:24:22.589 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.535+0000 7f18416f3700 1 Processor -- start 2023-07-14T18:24:22.590 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 -- start start 2023-07-14T18:24:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c0ff830 0x7f183c100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:22.591 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c103f30 0x7f183c101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c1a4140 con 0x7f183c0fee90 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c1a42b0 con 0x7f183c0ff830 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f18416f3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183c1a4470 con 0x7f183c103f30 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47946/0 (socket says 172.21.15.64:47946) 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 -- 172.21.15.64:0/438589122 learned_addr learned my addr 172.21.15.64:0/438589122 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:22.592 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183b7fe700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c103f30 0x7f183c101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183a7fc700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c0ff830 0x7f183c100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 -- 172.21.15.64:0/438589122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c103f30 msgr2=0x7f183c101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c103f30 0x7f183c101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 -- 172.21.15.64:0/438589122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c0ff830 msgr2=0x7f183c100de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c0ff830 0x7f183c100de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.536+0000 7f183affd700 1 -- 172.21.15.64:0/438589122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300095f0 con 0x7f183c0fee90 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f183affd700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1830009f50 tx=0x7f1830004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:22.593 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183001d070 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183c1a46f0 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f18300054c0 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1830003ab0 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.537+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183c1a4b90 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.538+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183c108110 con 0x7f183c0fee90 2023-07-14T18:24:22.594 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.539+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f18300047c0 con 0x7f183c0fee90 2023-07-14T18:24:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.540+0000 7f1823fff700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 0x7f18240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.540+0000 7f183a7fc700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 0x7f18240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.540+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f18300cdb20 con 0x7f183c0fee90 2023-07-14T18:24:22.595 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.540+0000 7f183a7fc700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 0x7f18240a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f183c102090 tx=0x7f182c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:22.596 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.544+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1830096e90 con 0x7f183c0fee90 2023-07-14T18:24:22.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.735+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f183c060a80 con 0x7f18240a11e0 2023-07-14T18:24:22.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.750+0000 7f1823fff700 1 -- 172.21.15.64:0/438589122 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426910 (secure 0 0 0) 0x7f183c060a80 con 0x7f18240a11e0 2023-07-14T18:24:22.807 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:22.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 msgr2=0x7f18240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:22.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 0x7f18240a3690 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f183c102090 tx=0x7f182c00b410 comp rx=0 tx=0).stop 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 msgr2=0x7f183c1008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1830009f50 tx=0x7f1830004710 comp rx=0 tx=0).stop 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 shutdown_connections 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f183c103f30 0x7f183c101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f18240a11e0 0x7f18240a3690 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f183c0fee90 0x7f183c1008a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 --2- 172.21.15.64:0/438589122 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f183c0ff830 0x7f183c100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:22.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.753+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 >> 172.21.15.64:0/438589122 conn(0x7f183c075ae0 msgr2=0x7f183c102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:22.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.754+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 shutdown_connections 2023-07-14T18:24:22.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:22.754+0000 7f18416f3700 1 -- 172.21.15.64:0/438589122 wait complete. 2023-07-14T18:24:22.811 INFO:teuthology.orchestra.run.smithi064.stderr:dumped all 2023-07-14T18:24:23.152 INFO:teuthology.orchestra.run.smithi064.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2023-07-14T18:24:21.477967+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656175,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":671,"num_read_kb":455,"num_write":417,"num_write_kb":3072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":345,"ondisk_log_size":345,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337852,"kb_used_data":13580,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124603972,"statfs":{"total":1151940427776,"available":1151594467328,"internally_reserved":0,"allocated":13905920,"data_stored":7662947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":105,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003600"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405474+0000","last_change":"2023-07-14T18:23:05.251113+0000","last_active":"2023-07-14T18:24:07.405474+0000","last_peered":"2023-07-14T18:24:07.405474+0000","last_clean":"2023-07-14T18:24:07.405474+0000","last_became_active":"2023-07-14T18:23:05.250773+0000","last_became_peered":"2023-07-14T18:23:05.250773+0000","last_unstale":"2023-07-14T18:24:07.405474+0000","last_undegraded":"2023-07-14T18:24:07.405474+0000","last_fullsized":"2023-07-14T18:24:07.405474+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:24:11.348750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405925+0000","last_change":"2023-07-14T18:23:05.252044+0000","last_active":"2023-07-14T18:24:07.405925+0000","last_peered":"2023-07-14T18:24:07.405925+0000","last_clean":"2023-07-14T18:24:07.405925+0000","last_became_active":"2023-07-14T18:23:05.251730+0000","last_became_peered":"2023-07-14T18:23:05.251730+0000","last_unstale":"2023-07-14T18:24:07.405925+0000","last_undegraded":"2023-07-14T18:24:07.405925+0000","last_fullsized":"2023-07-14T18:24:07.405925+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:08.123484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405720+0000","last_change":"2023-07-14T18:23:05.250367+0000","last_active":"2023-07-14T18:24:07.405720+0000","last_peered":"2023-07-14T18:24:07.405720+0000","last_clean":"2023-07-14T18:24:07.405720+0000","last_became_active":"2023-07-14T18:23:05.250034+0000","last_became_peered":"2023-07-14T18:23:05.250034+0000","last_unstale":"2023-07-14T18:24:07.405720+0000","last_undegraded":"2023-07-14T18:24:07.405720+0000","last_fullsized":"2023-07-14T18:24:07.405720+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:18.329769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410214+0000","last_change":"2023-07-14T18:23:05.255493+0000","last_active":"2023-07-14T18:24:07.410214+0000","last_peered":"2023-07-14T18:24:07.410214+0000","last_clean":"2023-07-14T18:24:07.410214+0000","last_became_active":"2023-07-14T18:23:05.254788+0000","last_became_peered":"2023-07-14T18:23:05.254788+0000","last_unstale":"2023-07-14T18:24:07.410214+0000","last_undegraded":"2023-07-14T18:24:07.410214+0000","last_fullsized":"2023-07-14T18:24:07.410214+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:35.873720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403632+0000","last_change":"2023-07-14T18:23:05.246551+0000","last_active":"2023-07-14T18:24:07.403632+0000","last_peered":"2023-07-14T18:24:07.403632+0000","last_clean":"2023-07-14T18:24:07.403632+0000","last_became_active":"2023-07-14T18:23:05.246248+0000","last_became_peered":"2023-07-14T18:23:05.246248+0000","last_unstale":"2023-07-14T18:24:07.403632+0000","last_undegraded":"2023-07-14T18:24:07.403632+0000","last_fullsized":"2023-07-14T18:24:07.403632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:47.188696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599111+0000","last_change":"2023-07-14T18:23:05.248544+0000","last_active":"2023-07-14T18:24:07.599111+0000","last_peered":"2023-07-14T18:24:07.599111+0000","last_clean":"2023-07-14T18:24:07.599111+0000","last_became_active":"2023-07-14T18:23:05.248229+0000","last_became_peered":"2023-07-14T18:23:05.248229+0000","last_unstale":"2023-07-14T18:24:07.599111+0000","last_undegraded":"2023-07-14T18:24:07.599111+0000","last_fullsized":"2023-07-14T18:24:07.599111+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:22:17.483400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405673+0000","last_change":"2023-07-14T18:23:05.245019+0000","last_active":"2023-07-14T18:24:07.405673+0000","last_peered":"2023-07-14T18:24:07.405673+0000","last_clean":"2023-07-14T18:24:07.405673+0000","last_became_active":"2023-07-14T18:23:05.242800+0000","last_became_peered":"2023-07-14T18:23:05.242800+0000","last_unstale":"2023-07-14T18:24:07.405673+0000","last_undegraded":"2023-07-14T18:24:07.405673+0000","last_fullsized":"2023-07-14T18:24:07.405673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:40:14.526417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597894+0000","last_change":"2023-07-14T18:23:05.237676+0000","last_active":"2023-07-14T18:24:07.597894+0000","last_peered":"2023-07-14T18:24:07.597894+0000","last_clean":"2023-07-14T18:24:07.597894+0000","last_became_active":"2023-07-14T18:23:05.237256+0000","last_became_peered":"2023-07-14T18:23:05.237256+0000","last_unstale":"2023-07-14T18:24:07.597894+0000","last_undegraded":"2023-07-14T18:24:07.597894+0000","last_fullsized":"2023-07-14T18:24:07.597894+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:47.842854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413213+0000","last_change":"2023-07-14T18:24:07.413213+0000","last_active":"2023-07-14T18:24:07.413213+0000","last_peered":"2023-07-14T18:24:07.413213+0000","last_clean":"2023-07-14T18:24:07.413213+0000","last_became_active":"2023-07-14T18:24:07.412865+0000","last_became_peered":"2023-07-14T18:24:07.412865+0000","last_unstale":"2023-07-14T18:24:07.413213+0000","last_undegraded":"2023-07-14T18:24:07.413213+0000","last_fullsized":"2023-07-14T18:24:07.413213+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:35:33.198078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,11],"acting":[1,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599383+0000","last_change":"2023-07-14T18:23:05.257209+0000","last_active":"2023-07-14T18:24:07.599383+0000","last_peered":"2023-07-14T18:24:07.599383+0000","last_clean":"2023-07-14T18:24:07.599383+0000","last_became_active":"2023-07-14T18:23:05.256890+0000","last_became_peered":"2023-07-14T18:23:05.256890+0000","last_unstale":"2023-07-14T18:24:07.599383+0000","last_undegraded":"2023-07-14T18:24:07.599383+0000","last_fullsized":"2023-07-14T18:24:07.599383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:33.854389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406490+0000","last_change":"2023-07-14T18:23:05.252704+0000","last_active":"2023-07-14T18:24:07.406490+0000","last_peered":"2023-07-14T18:24:07.406490+0000","last_clean":"2023-07-14T18:24:07.406490+0000","last_became_active":"2023-07-14T18:23:05.252005+0000","last_became_peered":"2023-07-14T18:23:05.252005+0000","last_unstale":"2023-07-14T18:24:07.406490+0000","last_undegraded":"2023-07-14T18:24:07.406490+0000","last_fullsized":"2023-07-14T18:24:07.406490+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:37:44.364403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410203+0000","last_change":"2023-07-14T18:23:05.253531+0000","last_active":"2023-07-14T18:24:07.410203+0000","last_peered":"2023-07-14T18:24:07.410203+0000","last_clean":"2023-07-14T18:24:07.410203+0000","last_became_active":"2023-07-14T18:23:05.253229+0000","last_became_peered":"2023-07-14T18:23:05.253229+0000","last_unstale":"2023-07-14T18:24:07.410203+0000","last_undegraded":"2023-07-14T18:24:07.410203+0000","last_fullsized":"2023-07-14T18:24:07.410203+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:33.517728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820341+0000","last_change":"2023-07-14T18:23:05.247023+0000","last_active":"2023-07-14T18:24:07.820341+0000","last_peered":"2023-07-14T18:24:07.820341+0000","last_clean":"2023-07-14T18:24:07.820341+0000","last_became_active":"2023-07-14T18:23:05.246674+0000","last_became_peered":"2023-07-14T18:23:05.246674+0000","last_unstale":"2023-07-14T18:24:07.820341+0000","last_undegraded":"2023-07-14T18:24:07.820341+0000","last_fullsized":"2023-07-14T18:24:07.820341+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:48:47.802767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405516+0000","last_change":"2023-07-14T18:23:05.245843+0000","last_active":"2023-07-14T18:24:07.405516+0000","last_peered":"2023-07-14T18:24:07.405516+0000","last_clean":"2023-07-14T18:24:07.405516+0000","last_became_active":"2023-07-14T18:23:05.245020+0000","last_became_peered":"2023-07-14T18:23:05.245020+0000","last_unstale":"2023-07-14T18:24:07.405516+0000","last_undegraded":"2023-07-14T18:24:07.405516+0000","last_fullsized":"2023-07-14T18:24:07.405516+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:48:51.690671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405517+0000","last_change":"2023-07-14T18:23:05.255992+0000","last_active":"2023-07-14T18:24:07.405517+0000","last_peered":"2023-07-14T18:24:07.405517+0000","last_clean":"2023-07-14T18:24:07.405517+0000","last_became_active":"2023-07-14T18:23:05.255693+0000","last_became_peered":"2023-07-14T18:23:05.255693+0000","last_unstale":"2023-07-14T18:24:07.405517+0000","last_undegraded":"2023-07-14T18:24:07.405517+0000","last_fullsized":"2023-07-14T18:24:07.405517+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:28:59.951987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407820+0000","last_change":"2023-07-14T18:23:05.251875+0000","last_active":"2023-07-14T18:24:07.407820+0000","last_peered":"2023-07-14T18:24:07.407820+0000","last_clean":"2023-07-14T18:24:07.407820+0000","last_became_active":"2023-07-14T18:23:05.251568+0000","last_became_peered":"2023-07-14T18:23:05.251568+0000","last_unstale":"2023-07-14T18:24:07.407820+0000","last_undegraded":"2023-07-14T18:24:07.407820+0000","last_fullsized":"2023-07-14T18:24:07.407820+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:19.271529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405475+0000","last_change":"2023-07-14T18:23:05.251240+0000","last_active":"2023-07-14T18:24:07.405475+0000","last_peered":"2023-07-14T18:24:07.405475+0000","last_clean":"2023-07-14T18:24:07.405475+0000","last_became_active":"2023-07-14T18:23:05.250949+0000","last_became_peered":"2023-07-14T18:23:05.250949+0000","last_unstale":"2023-07-14T18:24:07.405475+0000","last_undegraded":"2023-07-14T18:24:07.405475+0000","last_fullsized":"2023-07-14T18:24:07.405475+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:27:22.438148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407722+0000","last_change":"2023-07-14T18:23:05.241139+0000","last_active":"2023-07-14T18:24:07.407722+0000","last_peered":"2023-07-14T18:24:07.407722+0000","last_clean":"2023-07-14T18:24:07.407722+0000","last_became_active":"2023-07-14T18:23:05.240856+0000","last_became_peered":"2023-07-14T18:23:05.240856+0000","last_unstale":"2023-07-14T18:24:07.407722+0000","last_undegraded":"2023-07-14T18:24:07.407722+0000","last_fullsized":"2023-07-14T18:24:07.407722+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:48:18.810920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407624+0000","last_change":"2023-07-14T18:23:05.248109+0000","last_active":"2023-07-14T18:24:07.407624+0000","last_peered":"2023-07-14T18:24:07.407624+0000","last_clean":"2023-07-14T18:24:07.407624+0000","last_became_active":"2023-07-14T18:23:05.247634+0000","last_became_peered":"2023-07-14T18:23:05.247634+0000","last_unstale":"2023-07-14T18:24:07.407624+0000","last_undegraded":"2023-07-14T18:24:07.407624+0000","last_fullsized":"2023-07-14T18:24:07.407624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:43.490417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406996+0000","last_change":"2023-07-14T18:23:09.456276+0000","last_active":"2023-07-14T18:24:07.406996+0000","last_peered":"2023-07-14T18:24:07.406996+0000","last_clean":"2023-07-14T18:24:07.406996+0000","last_became_active":"2023-07-14T18:23:09.455938+0000","last_became_peered":"2023-07-14T18:23:09.455938+0000","last_unstale":"2023-07-14T18:24:07.406996+0000","last_undegraded":"2023-07-14T18:24:07.406996+0000","last_fullsized":"2023-07-14T18:24:07.406996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:27:08.177654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'13","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:16.366099+0000","last_change":"2023-07-14T18:23:04.419957+0000","last_active":"2023-07-14T18:24:16.366099+0000","last_peered":"2023-07-14T18:24:16.366099+0000","last_clean":"2023-07-14T18:24:16.366099+0000","last_became_active":"2023-07-14T18:23:04.419613+0000","last_became_peered":"2023-07-14T18:23:04.419613+0000","last_unstale":"2023-07-14T18:24:16.366099+0000","last_undegraded":"2023-07-14T18:24:16.366099+0000","last_fullsized":"2023-07-14T18:24:16.366099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:08:37.412855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":51,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598331+0000","last_change":"2023-07-14T18:23:05.253838+0000","last_active":"2023-07-14T18:24:07.598331+0000","last_peered":"2023-07-14T18:24:07.598331+0000","last_clean":"2023-07-14T18:24:07.598331+0000","last_became_active":"2023-07-14T18:23:05.253616+0000","last_became_peered":"2023-07-14T18:23:05.253616+0000","last_unstale":"2023-07-14T18:24:07.598331+0000","last_undegraded":"2023-07-14T18:24:07.598331+0000","last_fullsized":"2023-07-14T18:24:07.598331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:51.649314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409478+0000","last_change":"2023-07-14T18:23:09.456477+0000","last_active":"2023-07-14T18:24:07.409478+0000","last_peered":"2023-07-14T18:24:07.409478+0000","last_clean":"2023-07-14T18:24:07.409478+0000","last_became_active":"2023-07-14T18:23:09.456114+0000","last_became_peered":"2023-07-14T18:23:09.456114+0000","last_unstale":"2023-07-14T18:24:07.409478+0000","last_undegraded":"2023-07-14T18:24:07.409478+0000","last_fullsized":"2023-07-14T18:24:07.409478+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:24:08.585476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598686+0000","last_change":"2023-07-14T18:23:04.425595+0000","last_active":"2023-07-14T18:24:07.598686+0000","last_peered":"2023-07-14T18:24:07.598686+0000","last_clean":"2023-07-14T18:24:07.598686+0000","last_became_active":"2023-07-14T18:23:04.425251+0000","last_became_peered":"2023-07-14T18:23:04.425251+0000","last_unstale":"2023-07-14T18:24:07.598686+0000","last_undegraded":"2023-07-14T18:24:07.598686+0000","last_fullsized":"2023-07-14T18:24:07.598686+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:41:43.150320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405997+0000","last_change":"2023-07-14T18:23:05.250410+0000","last_active":"2023-07-14T18:24:07.405997+0000","last_peered":"2023-07-14T18:24:07.405997+0000","last_clean":"2023-07-14T18:24:07.405997+0000","last_became_active":"2023-07-14T18:23:05.250075+0000","last_became_peered":"2023-07-14T18:23:05.250075+0000","last_unstale":"2023-07-14T18:24:07.405997+0000","last_undegraded":"2023-07-14T18:24:07.405997+0000","last_fullsized":"2023-07-14T18:24:07.405997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:37:28.657481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409648+0000","last_change":"2023-07-14T18:23:09.457265+0000","last_active":"2023-07-14T18:24:07.409648+0000","last_peered":"2023-07-14T18:24:07.409648+0000","last_clean":"2023-07-14T18:24:07.409648+0000","last_became_active":"2023-07-14T18:23:09.456827+0000","last_became_peered":"2023-07-14T18:23:09.456827+0000","last_unstale":"2023-07-14T18:24:07.409648+0000","last_undegraded":"2023-07-14T18:24:07.409648+0000","last_fullsized":"2023-07-14T18:24:07.409648+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:20.527639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'12","reported_seq":37,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403960+0000","last_change":"2023-07-14T18:23:04.425035+0000","last_active":"2023-07-14T18:24:07.403960+0000","last_peered":"2023-07-14T18:24:07.403960+0000","last_clean":"2023-07-14T18:24:07.403960+0000","last_became_active":"2023-07-14T18:23:04.424576+0000","last_became_peered":"2023-07-14T18:23:04.424576+0000","last_unstale":"2023-07-14T18:24:07.403960+0000","last_undegraded":"2023-07-14T18:24:07.403960+0000","last_fullsized":"2023-07-14T18:24:07.403960+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:43.378883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408067+0000","last_change":"2023-07-14T18:23:05.246369+0000","last_active":"2023-07-14T18:24:07.408067+0000","last_peered":"2023-07-14T18:24:07.408067+0000","last_clean":"2023-07-14T18:24:07.408067+0000","last_became_active":"2023-07-14T18:23:05.246044+0000","last_became_peered":"2023-07-14T18:23:05.246044+0000","last_unstale":"2023-07-14T18:24:07.408067+0000","last_undegraded":"2023-07-14T18:24:07.408067+0000","last_fullsized":"2023-07-14T18:24:07.408067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:17:27.452672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409589+0000","last_change":"2023-07-14T18:23:09.456473+0000","last_active":"2023-07-14T18:24:07.409589+0000","last_peered":"2023-07-14T18:24:07.409589+0000","last_clean":"2023-07-14T18:24:07.409589+0000","last_became_active":"2023-07-14T18:23:09.456057+0000","last_became_peered":"2023-07-14T18:23:09.456057+0000","last_unstale":"2023-07-14T18:24:07.409589+0000","last_undegraded":"2023-07-14T18:24:07.409589+0000","last_fullsized":"2023-07-14T18:24:07.409589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:47:26.524872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"85'5","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403362+0000","last_change":"2023-07-14T18:23:04.418302+0000","last_active":"2023-07-14T18:24:07.403362+0000","last_peered":"2023-07-14T18:24:07.403362+0000","last_clean":"2023-07-14T18:24:07.403362+0000","last_became_active":"2023-07-14T18:23:04.417959+0000","last_became_peered":"2023-07-14T18:23:04.417959+0000","last_unstale":"2023-07-14T18:24:07.403362+0000","last_undegraded":"2023-07-14T18:24:07.403362+0000","last_fullsized":"2023-07-14T18:24:07.403362+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:47.113574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409769+0000","last_change":"2023-07-14T18:23:05.251716+0000","last_active":"2023-07-14T18:24:07.409769+0000","last_peered":"2023-07-14T18:24:07.409769+0000","last_clean":"2023-07-14T18:24:07.409769+0000","last_became_active":"2023-07-14T18:23:05.251326+0000","last_became_peered":"2023-07-14T18:23:05.251326+0000","last_unstale":"2023-07-14T18:24:07.409769+0000","last_undegraded":"2023-07-14T18:24:07.409769+0000","last_fullsized":"2023-07-14T18:24:07.409769+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:07.113007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.600335+0000","last_change":"2023-07-14T18:24:07.600335+0000","last_active":"2023-07-14T18:24:07.600335+0000","last_peered":"2023-07-14T18:24:07.600335+0000","last_clean":"2023-07-14T18:24:07.600335+0000","last_became_active":"2023-07-14T18:24:07.599841+0000","last_became_peered":"2023-07-14T18:24:07.599841+0000","last_unstale":"2023-07-14T18:24:07.600335+0000","last_undegraded":"2023-07-14T18:24:07.600335+0000","last_fullsized":"2023-07-14T18:24:07.600335+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:34.685911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,7,11],"acting":[2,10,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:16.386864+0000","last_change":"2023-07-14T18:23:04.423341+0000","last_active":"2023-07-14T18:24:16.386864+0000","last_peered":"2023-07-14T18:24:16.386864+0000","last_clean":"2023-07-14T18:24:16.386864+0000","last_became_active":"2023-07-14T18:23:04.423017+0000","last_became_peered":"2023-07-14T18:23:04.423017+0000","last_unstale":"2023-07-14T18:24:16.386864+0000","last_undegraded":"2023-07-14T18:24:16.386864+0000","last_fullsized":"2023-07-14T18:24:16.386864+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:31.697182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403600+0000","last_change":"2023-07-14T18:23:05.244406+0000","last_active":"2023-07-14T18:24:07.403600+0000","last_peered":"2023-07-14T18:24:07.403600+0000","last_clean":"2023-07-14T18:24:07.403600+0000","last_became_active":"2023-07-14T18:23:05.244097+0000","last_became_peered":"2023-07-14T18:23:05.244097+0000","last_unstale":"2023-07-14T18:24:07.403600+0000","last_undegraded":"2023-07-14T18:24:07.403600+0000","last_fullsized":"2023-07-14T18:24:07.403600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:44.359944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407774+0000","last_change":"2023-07-14T18:23:09.449544+0000","last_active":"2023-07-14T18:24:07.407774+0000","last_peered":"2023-07-14T18:24:07.407774+0000","last_clean":"2023-07-14T18:24:07.407774+0000","last_became_active":"2023-07-14T18:23:09.449157+0000","last_became_peered":"2023-07-14T18:23:09.449157+0000","last_unstale":"2023-07-14T18:24:07.407774+0000","last_undegraded":"2023-07-14T18:24:07.407774+0000","last_fullsized":"2023-07-14T18:24:07.407774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:14:27.518543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'7","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599020+0000","last_change":"2023-07-14T18:23:04.424515+0000","last_active":"2023-07-14T18:24:07.599020+0000","last_peered":"2023-07-14T18:24:07.599020+0000","last_clean":"2023-07-14T18:24:07.599020+0000","last_became_active":"2023-07-14T18:23:04.424195+0000","last_became_peered":"2023-07-14T18:23:04.424195+0000","last_unstale":"2023-07-14T18:24:07.599020+0000","last_undegraded":"2023-07-14T18:24:07.599020+0000","last_fullsized":"2023-07-14T18:24:07.599020+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:57:13.399677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.822326+0000","last_change":"2023-07-14T18:24:07.822326+0000","last_active":"2023-07-14T18:24:07.822326+0000","last_peered":"2023-07-14T18:24:07.822326+0000","last_clean":"2023-07-14T18:24:07.822326+0000","last_became_active":"2023-07-14T18:24:07.821961+0000","last_became_peered":"2023-07-14T18:24:07.821961+0000","last_unstale":"2023-07-14T18:24:07.822326+0000","last_undegraded":"2023-07-14T18:24:07.822326+0000","last_fullsized":"2023-07-14T18:24:07.822326+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:49.138102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405293+0000","last_change":"2023-07-14T18:23:09.455157+0000","last_active":"2023-07-14T18:24:07.405293+0000","last_peered":"2023-07-14T18:24:07.405293+0000","last_clean":"2023-07-14T18:24:07.405293+0000","last_became_active":"2023-07-14T18:23:09.454617+0000","last_became_peered":"2023-07-14T18:23:09.454617+0000","last_unstale":"2023-07-14T18:24:07.405293+0000","last_undegraded":"2023-07-14T18:24:07.405293+0000","last_fullsized":"2023-07-14T18:24:07.405293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:36.010004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820213+0000","last_change":"2023-07-14T18:23:04.411633+0000","last_active":"2023-07-14T18:24:07.820213+0000","last_peered":"2023-07-14T18:24:07.820213+0000","last_clean":"2023-07-14T18:24:07.820213+0000","last_became_active":"2023-07-14T18:23:04.411269+0000","last_became_peered":"2023-07-14T18:23:04.411269+0000","last_unstale":"2023-07-14T18:24:07.820213+0000","last_undegraded":"2023-07-14T18:24:07.820213+0000","last_fullsized":"2023-07-14T18:24:07.820213+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:32.445990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598031+0000","last_change":"2023-07-14T18:23:05.243039+0000","last_active":"2023-07-14T18:24:07.598031+0000","last_peered":"2023-07-14T18:24:07.598031+0000","last_clean":"2023-07-14T18:24:07.598031+0000","last_became_active":"2023-07-14T18:23:05.242731+0000","last_became_peered":"2023-07-14T18:23:05.242731+0000","last_unstale":"2023-07-14T18:24:07.598031+0000","last_undegraded":"2023-07-14T18:24:07.598031+0000","last_fullsized":"2023-07-14T18:24:07.598031+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:06:38.928448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405774+0000","last_change":"2023-07-14T18:23:09.453060+0000","last_active":"2023-07-14T18:24:07.405774+0000","last_peered":"2023-07-14T18:24:07.405774+0000","last_clean":"2023-07-14T18:24:07.405774+0000","last_became_active":"2023-07-14T18:23:09.452653+0000","last_became_peered":"2023-07-14T18:23:09.452653+0000","last_unstale":"2023-07-14T18:24:07.405774+0000","last_undegraded":"2023-07-14T18:24:07.405774+0000","last_fullsized":"2023-07-14T18:24:07.405774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:31.966181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410048+0000","last_change":"2023-07-14T18:23:04.420438+0000","last_active":"2023-07-14T18:24:07.410048+0000","last_peered":"2023-07-14T18:24:07.410048+0000","last_clean":"2023-07-14T18:24:07.410048+0000","last_became_active":"2023-07-14T18:23:04.420119+0000","last_became_peered":"2023-07-14T18:23:04.420119+0000","last_unstale":"2023-07-14T18:24:07.410048+0000","last_undegraded":"2023-07-14T18:24:07.410048+0000","last_fullsized":"2023-07-14T18:24:07.410048+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:40:18.846973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407999+0000","last_change":"2023-07-14T18:23:05.240810+0000","last_active":"2023-07-14T18:24:07.407999+0000","last_peered":"2023-07-14T18:24:07.407999+0000","last_clean":"2023-07-14T18:24:07.407999+0000","last_became_active":"2023-07-14T18:23:05.240489+0000","last_became_peered":"2023-07-14T18:23:05.240489+0000","last_unstale":"2023-07-14T18:24:07.407999+0000","last_undegraded":"2023-07-14T18:24:07.407999+0000","last_fullsized":"2023-07-14T18:24:07.407999+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:36:19.820805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597965+0000","last_change":"2023-07-14T18:23:09.452262+0000","last_active":"2023-07-14T18:24:07.597965+0000","last_peered":"2023-07-14T18:24:07.597965+0000","last_clean":"2023-07-14T18:24:07.597965+0000","last_became_active":"2023-07-14T18:23:09.451882+0000","last_became_peered":"2023-07-14T18:23:09.451882+0000","last_unstale":"2023-07-14T18:24:07.597965+0000","last_undegraded":"2023-07-14T18:24:07.597965+0000","last_fullsized":"2023-07-14T18:24:07.597965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:40.703098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'4","reported_seq":128,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:16.365361+0000","last_change":"2023-07-14T18:23:04.422122+0000","last_active":"2023-07-14T18:24:16.365361+0000","last_peered":"2023-07-14T18:24:16.365361+0000","last_clean":"2023-07-14T18:24:16.365361+0000","last_became_active":"2023-07-14T18:23:04.421805+0000","last_became_peered":"2023-07-14T18:23:04.421805+0000","last_unstale":"2023-07-14T18:24:16.365361+0000","last_undegraded":"2023-07-14T18:24:16.365361+0000","last_fullsized":"2023-07-14T18:24:16.365361+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:24.616145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408207+0000","last_change":"2023-07-14T18:23:05.242861+0000","last_active":"2023-07-14T18:24:07.408207+0000","last_peered":"2023-07-14T18:24:07.408207+0000","last_clean":"2023-07-14T18:24:07.408207+0000","last_became_active":"2023-07-14T18:23:05.242542+0000","last_became_peered":"2023-07-14T18:23:05.242542+0000","last_unstale":"2023-07-14T18:24:07.408207+0000","last_undegraded":"2023-07-14T18:24:07.408207+0000","last_fullsized":"2023-07-14T18:24:07.408207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:44:26.421835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413138+0000","last_change":"2023-07-14T18:24:07.412404+0000","last_active":"2023-07-14T18:24:07.413138+0000","last_peered":"2023-07-14T18:24:07.413138+0000","last_clean":"2023-07-14T18:24:07.413138+0000","last_became_active":"2023-07-14T18:24:07.411933+0000","last_became_peered":"2023-07-14T18:24:07.411933+0000","last_unstale":"2023-07-14T18:24:07.413138+0000","last_undegraded":"2023-07-14T18:24:07.413138+0000","last_fullsized":"2023-07-14T18:24:07.413138+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:22.236815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406710+0000","last_change":"2023-07-14T18:23:04.418340+0000","last_active":"2023-07-14T18:24:07.406710+0000","last_peered":"2023-07-14T18:24:07.406710+0000","last_clean":"2023-07-14T18:24:07.406710+0000","last_became_active":"2023-07-14T18:23:04.417965+0000","last_became_peered":"2023-07-14T18:23:04.417965+0000","last_unstale":"2023-07-14T18:24:07.406710+0000","last_undegraded":"2023-07-14T18:24:07.406710+0000","last_fullsized":"2023-07-14T18:24:07.406710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:32:56.571997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.411973+0000","last_change":"2023-07-14T18:24:07.411973+0000","last_active":"2023-07-14T18:24:07.411973+0000","last_peered":"2023-07-14T18:24:07.411973+0000","last_clean":"2023-07-14T18:24:07.411973+0000","last_became_active":"2023-07-14T18:24:07.411638+0000","last_became_peered":"2023-07-14T18:24:07.411638+0000","last_unstale":"2023-07-14T18:24:07.411973+0000","last_undegraded":"2023-07-14T18:24:07.411973+0000","last_fullsized":"2023-07-14T18:24:07.411973+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:28.531130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413964+0000","last_change":"2023-07-14T18:24:07.413964+0000","last_active":"2023-07-14T18:24:07.413964+0000","last_peered":"2023-07-14T18:24:07.413964+0000","last_clean":"2023-07-14T18:24:07.413964+0000","last_became_active":"2023-07-14T18:24:07.413507+0000","last_became_peered":"2023-07-14T18:24:07.413507+0000","last_unstale":"2023-07-14T18:24:07.413964+0000","last_undegraded":"2023-07-14T18:24:07.413964+0000","last_fullsized":"2023-07-14T18:24:07.413964+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:13:27.116037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409637+0000","last_change":"2023-07-14T18:23:04.427271+0000","last_active":"2023-07-14T18:24:07.409637+0000","last_peered":"2023-07-14T18:24:07.409637+0000","last_clean":"2023-07-14T18:24:07.409637+0000","last_became_active":"2023-07-14T18:23:04.426939+0000","last_became_peered":"2023-07-14T18:23:04.426939+0000","last_unstale":"2023-07-14T18:24:07.409637+0000","last_undegraded":"2023-07-14T18:24:07.409637+0000","last_fullsized":"2023-07-14T18:24:07.409637+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:10:25.197615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408340+0000","last_change":"2023-07-14T18:23:05.255028+0000","last_active":"2023-07-14T18:24:07.408340+0000","last_peered":"2023-07-14T18:24:07.408340+0000","last_clean":"2023-07-14T18:24:07.408340+0000","last_became_active":"2023-07-14T18:23:05.254417+0000","last_became_peered":"2023-07-14T18:23:05.254417+0000","last_unstale":"2023-07-14T18:24:07.408340+0000","last_undegraded":"2023-07-14T18:24:07.408340+0000","last_fullsized":"2023-07-14T18:24:07.408340+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:15:33.259444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407780+0000","last_change":"2023-07-14T18:23:09.454601+0000","last_active":"2023-07-14T18:24:07.407780+0000","last_peered":"2023-07-14T18:24:07.407780+0000","last_clean":"2023-07-14T18:24:07.407780+0000","last_became_active":"2023-07-14T18:23:09.454209+0000","last_became_peered":"2023-07-14T18:23:09.454209+0000","last_unstale":"2023-07-14T18:24:07.407780+0000","last_undegraded":"2023-07-14T18:24:07.407780+0000","last_fullsized":"2023-07-14T18:24:07.407780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:17:42.309537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'33","reported_seq":163,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:16.365373+0000","last_change":"2023-07-14T18:23:04.424141+0000","last_active":"2023-07-14T18:24:16.365373+0000","last_peered":"2023-07-14T18:24:16.365373+0000","last_clean":"2023-07-14T18:24:16.365373+0000","last_became_active":"2023-07-14T18:23:04.423397+0000","last_became_peered":"2023-07-14T18:23:04.423397+0000","last_unstale":"2023-07-14T18:24:16.365373+0000","last_undegraded":"2023-07-14T18:24:16.365373+0000","last_fullsized":"2023-07-14T18:24:16.365373+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:00.246915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58551,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":104,"num_read_kb":108,"num_write":35,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403773+0000","last_change":"2023-07-14T18:23:05.249509+0000","last_active":"2023-07-14T18:24:07.403773+0000","last_peered":"2023-07-14T18:24:07.403773+0000","last_clean":"2023-07-14T18:24:07.403773+0000","last_became_active":"2023-07-14T18:23:05.249237+0000","last_became_peered":"2023-07-14T18:23:05.249237+0000","last_unstale":"2023-07-14T18:24:07.403773+0000","last_undegraded":"2023-07-14T18:24:07.403773+0000","last_fullsized":"2023-07-14T18:24:07.403773+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:00:19.677064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410030+0000","last_change":"2023-07-14T18:23:09.453762+0000","last_active":"2023-07-14T18:24:07.410030+0000","last_peered":"2023-07-14T18:24:07.410030+0000","last_clean":"2023-07-14T18:24:07.410030+0000","last_became_active":"2023-07-14T18:23:09.453280+0000","last_became_peered":"2023-07-14T18:23:09.453280+0000","last_unstale":"2023-07-14T18:24:07.410030+0000","last_undegraded":"2023-07-14T18:24:07.410030+0000","last_fullsized":"2023-07-14T18:24:07.410030+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:16:15.831940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'9","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407198+0000","last_change":"2023-07-14T18:23:04.424639+0000","last_active":"2023-07-14T18:24:07.407198+0000","last_peered":"2023-07-14T18:24:07.407198+0000","last_clean":"2023-07-14T18:24:07.407198+0000","last_became_active":"2023-07-14T18:23:04.424156+0000","last_became_peered":"2023-07-14T18:23:04.424156+0000","last_unstale":"2023-07-14T18:24:07.407198+0000","last_undegraded":"2023-07-14T18:24:07.407198+0000","last_fullsized":"2023-07-14T18:24:07.407198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:03.898501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820320+0000","last_change":"2023-07-14T18:23:05.247903+0000","last_active":"2023-07-14T18:24:07.820320+0000","last_peered":"2023-07-14T18:24:07.820320+0000","last_clean":"2023-07-14T18:24:07.820320+0000","last_became_active":"2023-07-14T18:23:05.247590+0000","last_became_peered":"2023-07-14T18:23:05.247590+0000","last_unstale":"2023-07-14T18:24:07.820320+0000","last_undegraded":"2023-07-14T18:24:07.820320+0000","last_fullsized":"2023-07-14T18:24:07.820320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:28:38.028092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405119+0000","last_change":"2023-07-14T18:23:09.455142+0000","last_active":"2023-07-14T18:24:07.405119+0000","last_peered":"2023-07-14T18:24:07.405119+0000","last_clean":"2023-07-14T18:24:07.405119+0000","last_became_active":"2023-07-14T18:23:09.454706+0000","last_became_peered":"2023-07-14T18:23:09.454706+0000","last_unstale":"2023-07-14T18:24:07.405119+0000","last_undegraded":"2023-07-14T18:24:07.405119+0000","last_fullsized":"2023-07-14T18:24:07.405119+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:37:57.627580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'5","reported_seq":32,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409613+0000","last_change":"2023-07-14T18:23:04.423347+0000","last_active":"2023-07-14T18:24:07.409613+0000","last_peered":"2023-07-14T18:24:07.409613+0000","last_clean":"2023-07-14T18:24:07.409613+0000","last_became_active":"2023-07-14T18:23:04.423033+0000","last_became_peered":"2023-07-14T18:23:04.423033+0000","last_unstale":"2023-07-14T18:24:07.409613+0000","last_undegraded":"2023-07-14T18:24:07.409613+0000","last_fullsized":"2023-07-14T18:24:07.409613+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:13.349187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405811+0000","last_change":"2023-07-14T18:23:05.254564+0000","last_active":"2023-07-14T18:24:07.405811+0000","last_peered":"2023-07-14T18:24:07.405811+0000","last_clean":"2023-07-14T18:24:07.405811+0000","last_became_active":"2023-07-14T18:23:05.254219+0000","last_became_peered":"2023-07-14T18:23:05.254219+0000","last_unstale":"2023-07-14T18:24:07.405811+0000","last_undegraded":"2023-07-14T18:24:07.405811+0000","last_fullsized":"2023-07-14T18:24:07.405811+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:17:48.799901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407367+0000","last_change":"2023-07-14T18:23:09.455638+0000","last_active":"2023-07-14T18:24:07.407367+0000","last_peered":"2023-07-14T18:24:07.407367+0000","last_clean":"2023-07-14T18:24:07.407367+0000","last_became_active":"2023-07-14T18:23:09.455147+0000","last_became_peered":"2023-07-14T18:23:09.455147+0000","last_unstale":"2023-07-14T18:24:07.407367+0000","last_undegraded":"2023-07-14T18:24:07.407367+0000","last_fullsized":"2023-07-14T18:24:07.407367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:31:02.627246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'5","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:18.369385+0000","last_change":"2023-07-14T18:23:04.420483+0000","last_active":"2023-07-14T18:24:18.369385+0000","last_peered":"2023-07-14T18:24:18.369385+0000","last_clean":"2023-07-14T18:24:18.369385+0000","last_became_active":"2023-07-14T18:23:04.420154+0000","last_became_peered":"2023-07-14T18:23:04.420154+0000","last_unstale":"2023-07-14T18:24:18.369385+0000","last_undegraded":"2023-07-14T18:24:18.369385+0000","last_fullsized":"2023-07-14T18:24:18.369385+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:35:05.604630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405854+0000","last_change":"2023-07-14T18:23:05.249959+0000","last_active":"2023-07-14T18:24:07.405854+0000","last_peered":"2023-07-14T18:24:07.405854+0000","last_clean":"2023-07-14T18:24:07.405854+0000","last_became_active":"2023-07-14T18:23:05.249664+0000","last_became_peered":"2023-07-14T18:23:05.249664+0000","last_unstale":"2023-07-14T18:24:07.405854+0000","last_undegraded":"2023-07-14T18:24:07.405854+0000","last_fullsized":"2023-07-14T18:24:07.405854+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:01:16.422448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820996+0000","last_change":"2023-07-14T18:23:09.449699+0000","last_active":"2023-07-14T18:24:07.820996+0000","last_peered":"2023-07-14T18:24:07.820996+0000","last_clean":"2023-07-14T18:24:07.820996+0000","last_became_active":"2023-07-14T18:23:09.449324+0000","last_became_peered":"2023-07-14T18:23:09.449324+0000","last_unstale":"2023-07-14T18:24:07.820996+0000","last_undegraded":"2023-07-14T18:24:07.820996+0000","last_fullsized":"2023-07-14T18:24:07.820996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:03:00.885274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'17","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403130+0000","last_change":"2023-07-14T18:23:04.415693+0000","last_active":"2023-07-14T18:24:07.403130+0000","last_peered":"2023-07-14T18:24:07.403130+0000","last_clean":"2023-07-14T18:24:07.403130+0000","last_became_active":"2023-07-14T18:23:04.415450+0000","last_became_peered":"2023-07-14T18:23:04.415450+0000","last_unstale":"2023-07-14T18:24:07.403130+0000","last_undegraded":"2023-07-14T18:24:07.403130+0000","last_fullsized":"2023-07-14T18:24:07.403130+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:58:22.401795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413544+0000","last_change":"2023-07-14T18:24:07.413544+0000","last_active":"2023-07-14T18:24:07.413544+0000","last_peered":"2023-07-14T18:24:07.413544+0000","last_clean":"2023-07-14T18:24:07.413544+0000","last_became_active":"2023-07-14T18:24:07.413217+0000","last_became_peered":"2023-07-14T18:24:07.413217+0000","last_unstale":"2023-07-14T18:24:07.413544+0000","last_undegraded":"2023-07-14T18:24:07.413544+0000","last_fullsized":"2023-07-14T18:24:07.413544+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:50:24.792524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410040+0000","last_change":"2023-07-14T18:23:09.453584+0000","last_active":"2023-07-14T18:24:07.410040+0000","last_peered":"2023-07-14T18:24:07.410040+0000","last_clean":"2023-07-14T18:24:07.410040+0000","last_became_active":"2023-07-14T18:23:09.453107+0000","last_became_peered":"2023-07-14T18:23:09.453107+0000","last_unstale":"2023-07-14T18:24:07.410040+0000","last_undegraded":"2023-07-14T18:24:07.410040+0000","last_fullsized":"2023-07-14T18:24:07.410040+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:36:18.284526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404072+0000","last_change":"2023-07-14T18:23:04.424208+0000","last_active":"2023-07-14T18:24:07.404072+0000","last_peered":"2023-07-14T18:24:07.404072+0000","last_clean":"2023-07-14T18:24:07.404072+0000","last_became_active":"2023-07-14T18:23:04.423882+0000","last_became_peered":"2023-07-14T18:23:04.423882+0000","last_unstale":"2023-07-14T18:24:07.404072+0000","last_undegraded":"2023-07-14T18:24:07.404072+0000","last_fullsized":"2023-07-14T18:24:07.404072+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:32:44.906119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405246+0000","last_change":"2023-07-14T18:23:05.254716+0000","last_active":"2023-07-14T18:24:07.405246+0000","last_peered":"2023-07-14T18:24:07.405246+0000","last_clean":"2023-07-14T18:24:07.405246+0000","last_became_active":"2023-07-14T18:23:05.254401+0000","last_became_peered":"2023-07-14T18:23:05.254401+0000","last_unstale":"2023-07-14T18:24:07.405246+0000","last_undegraded":"2023-07-14T18:24:07.405246+0000","last_fullsized":"2023-07-14T18:24:07.405246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:25:41.007811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403367+0000","last_change":"2023-07-14T18:23:09.447009+0000","last_active":"2023-07-14T18:24:07.403367+0000","last_peered":"2023-07-14T18:24:07.403367+0000","last_clean":"2023-07-14T18:24:07.403367+0000","last_became_active":"2023-07-14T18:23:09.446584+0000","last_became_peered":"2023-07-14T18:23:09.446584+0000","last_unstale":"2023-07-14T18:24:07.403367+0000","last_undegraded":"2023-07-14T18:24:07.403367+0000","last_fullsized":"2023-07-14T18:24:07.403367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:06.743747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"85'1","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.413261+0000","last_change":"2023-07-14T18:24:07.413261+0000","last_active":"2023-07-14T18:24:07.413261+0000","last_peered":"2023-07-14T18:24:07.413261+0000","last_clean":"2023-07-14T18:24:07.413261+0000","last_became_active":"2023-07-14T18:24:07.412759+0000","last_became_peered":"2023-07-14T18:24:07.412759+0000","last_unstale":"2023-07-14T18:24:07.413261+0000","last_undegraded":"2023-07-14T18:24:07.413261+0000","last_fullsized":"2023-07-14T18:24:07.413261+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:21:09.725334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,0],"acting":[9,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599408+0000","last_change":"2023-07-14T18:23:05.245544+0000","last_active":"2023-07-14T18:24:07.599408+0000","last_peered":"2023-07-14T18:24:07.599408+0000","last_clean":"2023-07-14T18:24:07.599408+0000","last_became_active":"2023-07-14T18:23:05.245292+0000","last_became_peered":"2023-07-14T18:23:05.245292+0000","last_unstale":"2023-07-14T18:24:07.599408+0000","last_undegraded":"2023-07-14T18:24:07.599408+0000","last_fullsized":"2023-07-14T18:24:07.599408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:28:33.092745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820487+0000","last_change":"2023-07-14T18:23:09.451246+0000","last_active":"2023-07-14T18:24:07.820487+0000","last_peered":"2023-07-14T18:24:07.820487+0000","last_clean":"2023-07-14T18:24:07.820487+0000","last_became_active":"2023-07-14T18:23:09.450840+0000","last_became_peered":"2023-07-14T18:23:09.450840+0000","last_unstale":"2023-07-14T18:24:07.820487+0000","last_undegraded":"2023-07-14T18:24:07.820487+0000","last_fullsized":"2023-07-14T18:24:07.820487+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:09:47.913784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'5","reported_seq":81,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:17.367802+0000","last_change":"2023-07-14T18:23:04.425338+0000","last_active":"2023-07-14T18:24:17.367802+0000","last_peered":"2023-07-14T18:24:17.367802+0000","last_clean":"2023-07-14T18:24:17.367802+0000","last_became_active":"2023-07-14T18:23:04.424983+0000","last_became_peered":"2023-07-14T18:23:04.424983+0000","last_unstale":"2023-07-14T18:24:17.367802+0000","last_undegraded":"2023-07-14T18:24:17.367802+0000","last_fullsized":"2023-07-14T18:24:17.367802+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:29:23.957543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":540,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599127+0000","last_change":"2023-07-14T18:23:05.256804+0000","last_active":"2023-07-14T18:24:07.599127+0000","last_peered":"2023-07-14T18:24:07.599127+0000","last_clean":"2023-07-14T18:24:07.599127+0000","last_became_active":"2023-07-14T18:23:05.256380+0000","last_became_peered":"2023-07-14T18:23:05.256380+0000","last_unstale":"2023-07-14T18:24:07.599127+0000","last_undegraded":"2023-07-14T18:24:07.599127+0000","last_fullsized":"2023-07-14T18:24:07.599127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:22.907702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821088+0000","last_change":"2023-07-14T18:23:09.450381+0000","last_active":"2023-07-14T18:24:07.821088+0000","last_peered":"2023-07-14T18:24:07.821088+0000","last_clean":"2023-07-14T18:24:07.821088+0000","last_became_active":"2023-07-14T18:23:09.449949+0000","last_became_peered":"2023-07-14T18:23:09.449949+0000","last_unstale":"2023-07-14T18:24:07.821088+0000","last_undegraded":"2023-07-14T18:24:07.821088+0000","last_fullsized":"2023-07-14T18:24:07.821088+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:23:26.669137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410319+0000","last_change":"2023-07-14T18:23:05.249410+0000","last_active":"2023-07-14T18:24:07.410319+0000","last_peered":"2023-07-14T18:24:07.410319+0000","last_clean":"2023-07-14T18:24:07.410319+0000","last_became_active":"2023-07-14T18:23:05.249102+0000","last_became_peered":"2023-07-14T18:23:05.249102+0000","last_unstale":"2023-07-14T18:24:07.410319+0000","last_undegraded":"2023-07-14T18:24:07.410319+0000","last_fullsized":"2023-07-14T18:24:07.410319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:49:11.487403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'5","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:18.368651+0000","last_change":"2023-07-14T18:23:04.423851+0000","last_active":"2023-07-14T18:24:18.368651+0000","last_peered":"2023-07-14T18:24:18.368651+0000","last_clean":"2023-07-14T18:24:18.368651+0000","last_became_active":"2023-07-14T18:23:04.423484+0000","last_became_peered":"2023-07-14T18:23:04.423484+0000","last_unstale":"2023-07-14T18:24:18.368651+0000","last_undegraded":"2023-07-14T18:24:18.368651+0000","last_fullsized":"2023-07-14T18:24:18.368651+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:05:44.654417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406370+0000","last_change":"2023-07-14T18:23:09.447315+0000","last_active":"2023-07-14T18:24:07.406370+0000","last_peered":"2023-07-14T18:24:07.406370+0000","last_clean":"2023-07-14T18:24:07.406370+0000","last_became_active":"2023-07-14T18:23:09.446894+0000","last_became_peered":"2023-07-14T18:23:09.446894+0000","last_unstale":"2023-07-14T18:24:07.406370+0000","last_undegraded":"2023-07-14T18:24:07.406370+0000","last_fullsized":"2023-07-14T18:24:07.406370+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:01.561467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:05.250036+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:05.249717+0000","last_became_peered":"2023-07-14T18:23:05.249717+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:01.628715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'6","reported_seq":35,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409780+0000","last_change":"2023-07-14T18:23:04.418978+0000","last_active":"2023-07-14T18:24:07.409780+0000","last_peered":"2023-07-14T18:24:07.409780+0000","last_clean":"2023-07-14T18:24:07.409780+0000","last_became_active":"2023-07-14T18:23:04.418644+0000","last_became_peered":"2023-07-14T18:23:04.418644+0000","last_unstale":"2023-07-14T18:24:07.409780+0000","last_undegraded":"2023-07-14T18:24:07.409780+0000","last_fullsized":"2023-07-14T18:24:07.409780+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:45:44.875754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":348,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405706+0000","last_change":"2023-07-14T18:22:05.798027+0000","last_active":"2023-07-14T18:24:07.405706+0000","last_peered":"2023-07-14T18:24:07.405706+0000","last_clean":"2023-07-14T18:24:07.405706+0000","last_became_active":"2023-07-14T18:22:05.797666+0000","last_became_peered":"2023-07-14T18:22:05.797666+0000","last_unstale":"2023-07-14T18:24:07.405706+0000","last_undegraded":"2023-07-14T18:24:07.405706+0000","last_fullsized":"2023-07-14T18:24:07.405706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:19:14.476837+0000","last_clean_scrub_stamp":"2023-07-14T18:19:14.476837+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:46:07.000585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410311+0000","last_change":"2023-07-14T18:23:09.445018+0000","last_active":"2023-07-14T18:24:07.410311+0000","last_peered":"2023-07-14T18:24:07.410311+0000","last_clean":"2023-07-14T18:24:07.410311+0000","last_became_active":"2023-07-14T18:23:09.444601+0000","last_became_peered":"2023-07-14T18:23:09.444601+0000","last_unstale":"2023-07-14T18:24:07.410311+0000","last_undegraded":"2023-07-14T18:24:07.410311+0000","last_fullsized":"2023-07-14T18:24:07.410311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:49:22.806069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820058+0000","last_change":"2023-07-14T18:23:05.256205+0000","last_active":"2023-07-14T18:24:07.820058+0000","last_peered":"2023-07-14T18:24:07.820058+0000","last_clean":"2023-07-14T18:24:07.820058+0000","last_became_active":"2023-07-14T18:23:05.255872+0000","last_became_peered":"2023-07-14T18:23:05.255872+0000","last_unstale":"2023-07-14T18:24:07.820058+0000","last_undegraded":"2023-07-14T18:24:07.820058+0000","last_fullsized":"2023-07-14T18:24:07.820058+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:00.275183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'4","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405700+0000","last_change":"2023-07-14T18:23:04.426790+0000","last_active":"2023-07-14T18:24:07.405700+0000","last_peered":"2023-07-14T18:24:07.405700+0000","last_clean":"2023-07-14T18:24:07.405700+0000","last_became_active":"2023-07-14T18:23:04.426452+0000","last_became_peered":"2023-07-14T18:23:04.426452+0000","last_unstale":"2023-07-14T18:24:07.405700+0000","last_undegraded":"2023-07-14T18:24:07.405700+0000","last_fullsized":"2023-07-14T18:24:07.405700+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:58:04.719205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405561+0000","last_change":"2023-07-14T18:23:09.455275+0000","last_active":"2023-07-14T18:24:07.405561+0000","last_peered":"2023-07-14T18:24:07.405561+0000","last_clean":"2023-07-14T18:24:07.405561+0000","last_became_active":"2023-07-14T18:23:09.454926+0000","last_became_peered":"2023-07-14T18:23:09.454926+0000","last_unstale":"2023-07-14T18:24:07.405561+0000","last_undegraded":"2023-07-14T18:24:07.405561+0000","last_fullsized":"2023-07-14T18:24:07.405561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:20:44.840479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'4","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403787+0000","last_change":"2023-07-14T18:23:04.416067+0000","last_active":"2023-07-14T18:24:07.403787+0000","last_peered":"2023-07-14T18:24:07.403787+0000","last_clean":"2023-07-14T18:24:07.403787+0000","last_became_active":"2023-07-14T18:23:04.415678+0000","last_became_peered":"2023-07-14T18:23:04.415678+0000","last_unstale":"2023-07-14T18:24:07.403787+0000","last_undegraded":"2023-07-14T18:24:07.403787+0000","last_fullsized":"2023-07-14T18:24:07.403787+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:56:58.485409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820701+0000","last_change":"2023-07-14T18:23:05.251193+0000","last_active":"2023-07-14T18:24:07.820701+0000","last_peered":"2023-07-14T18:24:07.820701+0000","last_clean":"2023-07-14T18:24:07.820701+0000","last_became_active":"2023-07-14T18:23:05.250852+0000","last_became_peered":"2023-07-14T18:23:05.250852+0000","last_unstale":"2023-07-14T18:24:07.820701+0000","last_undegraded":"2023-07-14T18:24:07.820701+0000","last_fullsized":"2023-07-14T18:24:07.820701+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:52:09.481241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.597956+0000","last_change":"2023-07-14T18:23:09.458652+0000","last_active":"2023-07-14T18:24:07.597956+0000","last_peered":"2023-07-14T18:24:07.597956+0000","last_clean":"2023-07-14T18:24:07.597956+0000","last_became_active":"2023-07-14T18:23:09.456085+0000","last_became_peered":"2023-07-14T18:23:09.456085+0000","last_unstale":"2023-07-14T18:24:07.597956+0000","last_undegraded":"2023-07-14T18:24:07.597956+0000","last_fullsized":"2023-07-14T18:24:07.597956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:55.573942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'9","reported_seq":33,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407882+0000","last_change":"2023-07-14T18:23:04.418722+0000","last_active":"2023-07-14T18:24:07.407882+0000","last_peered":"2023-07-14T18:24:07.407882+0000","last_clean":"2023-07-14T18:24:07.407882+0000","last_became_active":"2023-07-14T18:23:04.417980+0000","last_became_peered":"2023-07-14T18:23:04.417980+0000","last_unstale":"2023-07-14T18:24:07.407882+0000","last_undegraded":"2023-07-14T18:24:07.407882+0000","last_fullsized":"2023-07-14T18:24:07.407882+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:55:27.571291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598511+0000","last_change":"2023-07-14T18:23:05.254496+0000","last_active":"2023-07-14T18:24:07.598511+0000","last_peered":"2023-07-14T18:24:07.598511+0000","last_clean":"2023-07-14T18:24:07.598511+0000","last_became_active":"2023-07-14T18:23:05.254147+0000","last_became_peered":"2023-07-14T18:23:05.254147+0000","last_unstale":"2023-07-14T18:24:07.598511+0000","last_undegraded":"2023-07-14T18:24:07.598511+0000","last_fullsized":"2023-07-14T18:24:07.598511+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:37:57.497511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406745+0000","last_change":"2023-07-14T18:23:09.453161+0000","last_active":"2023-07-14T18:24:07.406745+0000","last_peered":"2023-07-14T18:24:07.406745+0000","last_clean":"2023-07-14T18:24:07.406745+0000","last_became_active":"2023-07-14T18:23:09.452691+0000","last_became_peered":"2023-07-14T18:23:09.452691+0000","last_unstale":"2023-07-14T18:24:07.406745+0000","last_undegraded":"2023-07-14T18:24:07.406745+0000","last_fullsized":"2023-07-14T18:24:07.406745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:40.891887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'7","reported_seq":34,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410518+0000","last_change":"2023-07-14T18:23:04.420243+0000","last_active":"2023-07-14T18:24:07.410518+0000","last_peered":"2023-07-14T18:24:07.410518+0000","last_clean":"2023-07-14T18:24:07.410518+0000","last_became_active":"2023-07-14T18:23:04.419901+0000","last_became_peered":"2023-07-14T18:23:04.419901+0000","last_unstale":"2023-07-14T18:24:07.410518+0000","last_undegraded":"2023-07-14T18:24:07.410518+0000","last_fullsized":"2023-07-14T18:24:07.410518+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:54.251569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405928+0000","last_change":"2023-07-14T18:23:05.255209+0000","last_active":"2023-07-14T18:24:07.405928+0000","last_peered":"2023-07-14T18:24:07.405928+0000","last_clean":"2023-07-14T18:24:07.405928+0000","last_became_active":"2023-07-14T18:23:05.254868+0000","last_became_peered":"2023-07-14T18:23:05.254868+0000","last_unstale":"2023-07-14T18:24:07.405928+0000","last_undegraded":"2023-07-14T18:24:07.405928+0000","last_fullsized":"2023-07-14T18:24:07.405928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:56:46.633579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.599735+0000","last_change":"2023-07-14T18:24:07.599735+0000","last_active":"2023-07-14T18:24:07.599735+0000","last_peered":"2023-07-14T18:24:07.599735+0000","last_clean":"2023-07-14T18:24:07.599735+0000","last_became_active":"2023-07-14T18:24:07.599353+0000","last_became_peered":"2023-07-14T18:24:07.599353+0000","last_unstale":"2023-07-14T18:24:07.599735+0000","last_undegraded":"2023-07-14T18:24:07.599735+0000","last_fullsized":"2023-07-14T18:24:07.599735+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:36.613425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,11],"acting":[4,5,10,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407843+0000","last_change":"2023-07-14T18:23:04.417790+0000","last_active":"2023-07-14T18:24:07.407843+0000","last_peered":"2023-07-14T18:24:07.407843+0000","last_clean":"2023-07-14T18:24:07.407843+0000","last_became_active":"2023-07-14T18:23:04.417486+0000","last_became_peered":"2023-07-14T18:23:04.417486+0000","last_unstale":"2023-07-14T18:24:07.407843+0000","last_undegraded":"2023-07-14T18:24:07.407843+0000","last_fullsized":"2023-07-14T18:24:07.407843+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:27:45.050723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.403163+0000","last_change":"2023-07-14T18:23:05.244410+0000","last_active":"2023-07-14T18:24:07.403163+0000","last_peered":"2023-07-14T18:24:07.403163+0000","last_clean":"2023-07-14T18:24:07.403163+0000","last_became_active":"2023-07-14T18:23:05.244051+0000","last_became_peered":"2023-07-14T18:23:05.244051+0000","last_unstale":"2023-07-14T18:24:07.403163+0000","last_undegraded":"2023-07-14T18:24:07.403163+0000","last_fullsized":"2023-07-14T18:24:07.403163+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:15:37.710301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407589+0000","last_change":"2023-07-14T18:23:09.450511+0000","last_active":"2023-07-14T18:24:07.407589+0000","last_peered":"2023-07-14T18:24:07.407589+0000","last_clean":"2023-07-14T18:24:07.407589+0000","last_became_active":"2023-07-14T18:23:09.450115+0000","last_became_peered":"2023-07-14T18:23:09.450115+0000","last_unstale":"2023-07-14T18:24:07.407589+0000","last_undegraded":"2023-07-14T18:24:07.407589+0000","last_fullsized":"2023-07-14T18:24:07.407589+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:21:02.479091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'7","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598146+0000","last_change":"2023-07-14T18:23:04.422960+0000","last_active":"2023-07-14T18:24:07.598146+0000","last_peered":"2023-07-14T18:24:07.598146+0000","last_clean":"2023-07-14T18:24:07.598146+0000","last_became_active":"2023-07-14T18:23:04.422536+0000","last_became_peered":"2023-07-14T18:23:04.422536+0000","last_unstale":"2023-07-14T18:24:07.598146+0000","last_undegraded":"2023-07-14T18:24:07.598146+0000","last_fullsized":"2023-07-14T18:24:07.598146+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:37.977157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.404952+0000","last_change":"2023-07-14T18:23:05.252002+0000","last_active":"2023-07-14T18:24:07.404952+0000","last_peered":"2023-07-14T18:24:07.404952+0000","last_clean":"2023-07-14T18:24:07.404952+0000","last_became_active":"2023-07-14T18:23:05.251698+0000","last_became_peered":"2023-07-14T18:23:05.251698+0000","last_unstale":"2023-07-14T18:24:07.404952+0000","last_undegraded":"2023-07-14T18:24:07.404952+0000","last_fullsized":"2023-07-14T18:24:07.404952+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:32:37.080797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410069+0000","last_change":"2023-07-14T18:23:09.449623+0000","last_active":"2023-07-14T18:24:07.410069+0000","last_peered":"2023-07-14T18:24:07.410069+0000","last_clean":"2023-07-14T18:24:07.410069+0000","last_became_active":"2023-07-14T18:23:09.448498+0000","last_became_peered":"2023-07-14T18:23:09.448498+0000","last_unstale":"2023-07-14T18:24:07.410069+0000","last_undegraded":"2023-07-14T18:24:07.410069+0000","last_fullsized":"2023-07-14T18:24:07.410069+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:39:00.967275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'3","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407086+0000","last_change":"2023-07-14T18:23:04.425332+0000","last_active":"2023-07-14T18:24:07.407086+0000","last_peered":"2023-07-14T18:24:07.407086+0000","last_clean":"2023-07-14T18:24:07.407086+0000","last_became_active":"2023-07-14T18:23:04.425046+0000","last_became_peered":"2023-07-14T18:23:04.425046+0000","last_unstale":"2023-07-14T18:24:07.407086+0000","last_undegraded":"2023-07-14T18:24:07.407086+0000","last_fullsized":"2023-07-14T18:24:07.407086+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:39.830952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598357+0000","last_change":"2023-07-14T18:23:05.245896+0000","last_active":"2023-07-14T18:24:07.598357+0000","last_peered":"2023-07-14T18:24:07.598357+0000","last_clean":"2023-07-14T18:24:07.598357+0000","last_became_active":"2023-07-14T18:23:05.245597+0000","last_became_peered":"2023-07-14T18:23:05.245597+0000","last_unstale":"2023-07-14T18:24:07.598357+0000","last_undegraded":"2023-07-14T18:24:07.598357+0000","last_fullsized":"2023-07-14T18:24:07.598357+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:12.153021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409937+0000","last_change":"2023-07-14T18:23:09.448991+0000","last_active":"2023-07-14T18:24:07.409937+0000","last_peered":"2023-07-14T18:24:07.409937+0000","last_clean":"2023-07-14T18:24:07.409937+0000","last_became_active":"2023-07-14T18:23:09.448548+0000","last_became_peered":"2023-07-14T18:23:09.448548+0000","last_unstale":"2023-07-14T18:24:07.409937+0000","last_undegraded":"2023-07-14T18:24:07.409937+0000","last_fullsized":"2023-07-14T18:24:07.409937+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:05:41.469044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.416915+0000","last_change":"2023-07-14T18:24:07.416556+0000","last_active":"2023-07-14T18:24:07.416915+0000","last_peered":"2023-07-14T18:24:07.416915+0000","last_clean":"2023-07-14T18:24:07.416915+0000","last_became_active":"2023-07-14T18:24:07.408596+0000","last_became_peered":"2023-07-14T18:24:07.408596+0000","last_unstale":"2023-07-14T18:24:07.416915+0000","last_undegraded":"2023-07-14T18:24:07.416915+0000","last_fullsized":"2023-07-14T18:24:07.416915+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:02:10.303257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405892+0000","last_change":"2023-07-14T18:23:05.251883+0000","last_active":"2023-07-14T18:24:07.405892+0000","last_peered":"2023-07-14T18:24:07.405892+0000","last_clean":"2023-07-14T18:24:07.405892+0000","last_became_active":"2023-07-14T18:23:05.251554+0000","last_became_peered":"2023-07-14T18:23:05.251554+0000","last_unstale":"2023-07-14T18:24:07.405892+0000","last_undegraded":"2023-07-14T18:24:07.405892+0000","last_fullsized":"2023-07-14T18:24:07.405892+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:59:09.124887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407944+0000","last_change":"2023-07-14T18:23:09.447123+0000","last_active":"2023-07-14T18:24:07.407944+0000","last_peered":"2023-07-14T18:24:07.407944+0000","last_clean":"2023-07-14T18:24:07.407944+0000","last_became_active":"2023-07-14T18:23:09.446719+0000","last_became_peered":"2023-07-14T18:23:09.446719+0000","last_unstale":"2023-07-14T18:24:07.407944+0000","last_undegraded":"2023-07-14T18:24:07.407944+0000","last_fullsized":"2023-07-14T18:24:07.407944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:33.118185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'6","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:16.366090+0000","last_change":"2023-07-14T18:23:04.421198+0000","last_active":"2023-07-14T18:24:16.366090+0000","last_peered":"2023-07-14T18:24:16.366090+0000","last_clean":"2023-07-14T18:24:16.366090+0000","last_became_active":"2023-07-14T18:23:04.420966+0000","last_became_peered":"2023-07-14T18:23:04.420966+0000","last_unstale":"2023-07-14T18:24:16.366090+0000","last_undegraded":"2023-07-14T18:24:16.366090+0000","last_fullsized":"2023-07-14T18:24:16.366090+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:07:33.188924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1512,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406610+0000","last_change":"2023-07-14T18:23:05.252126+0000","last_active":"2023-07-14T18:24:07.406610+0000","last_peered":"2023-07-14T18:24:07.406610+0000","last_clean":"2023-07-14T18:24:07.406610+0000","last_became_active":"2023-07-14T18:23:05.251755+0000","last_became_peered":"2023-07-14T18:23:05.251755+0000","last_unstale":"2023-07-14T18:24:07.406610+0000","last_undegraded":"2023-07-14T18:24:07.406610+0000","last_fullsized":"2023-07-14T18:24:07.406610+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:25:28.457835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406508+0000","last_change":"2023-07-14T18:23:09.453721+0000","last_active":"2023-07-14T18:24:07.406508+0000","last_peered":"2023-07-14T18:24:07.406508+0000","last_clean":"2023-07-14T18:24:07.406508+0000","last_became_active":"2023-07-14T18:23:09.453282+0000","last_became_peered":"2023-07-14T18:23:09.453282+0000","last_unstale":"2023-07-14T18:24:07.406508+0000","last_undegraded":"2023-07-14T18:24:07.406508+0000","last_fullsized":"2023-07-14T18:24:07.406508+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:42:15.619889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.409875+0000","last_change":"2023-07-14T18:23:04.420227+0000","last_active":"2023-07-14T18:24:07.409875+0000","last_peered":"2023-07-14T18:24:07.409875+0000","last_clean":"2023-07-14T18:24:07.409875+0000","last_became_active":"2023-07-14T18:23:04.419889+0000","last_became_peered":"2023-07-14T18:23:04.419889+0000","last_unstale":"2023-07-14T18:24:07.409875+0000","last_undegraded":"2023-07-14T18:24:07.409875+0000","last_fullsized":"2023-07-14T18:24:07.409875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:21:05.734566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406230+0000","last_change":"2023-07-14T18:23:05.245766+0000","last_active":"2023-07-14T18:24:07.406230+0000","last_peered":"2023-07-14T18:24:07.406230+0000","last_clean":"2023-07-14T18:24:07.406230+0000","last_became_active":"2023-07-14T18:23:05.245455+0000","last_became_peered":"2023-07-14T18:23:05.245455+0000","last_unstale":"2023-07-14T18:24:07.406230+0000","last_undegraded":"2023-07-14T18:24:07.406230+0000","last_fullsized":"2023-07-14T18:24:07.406230+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:18.375201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408921+0000","last_change":"2023-07-14T18:23:09.448092+0000","last_active":"2023-07-14T18:24:07.408921+0000","last_peered":"2023-07-14T18:24:07.408921+0000","last_clean":"2023-07-14T18:24:07.408921+0000","last_became_active":"2023-07-14T18:23:09.447658+0000","last_became_peered":"2023-07-14T18:23:09.447658+0000","last_unstale":"2023-07-14T18:24:07.408921+0000","last_undegraded":"2023-07-14T18:24:07.408921+0000","last_fullsized":"2023-07-14T18:24:07.408921+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:08.423883+0000","last_clean_scrub_stamp":"2023-07-14T18:23:08.423883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:29:45.492125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.821139+0000","last_change":"2023-07-14T18:23:04.423023+0000","last_active":"2023-07-14T18:24:07.821139+0000","last_peered":"2023-07-14T18:24:07.821139+0000","last_clean":"2023-07-14T18:24:07.821139+0000","last_became_active":"2023-07-14T18:23:04.422709+0000","last_became_peered":"2023-07-14T18:23:04.422709+0000","last_unstale":"2023-07-14T18:24:07.821139+0000","last_undegraded":"2023-07-14T18:24:07.821139+0000","last_fullsized":"2023-07-14T18:24:07.821139+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:03.395009+0000","last_clean_scrub_stamp":"2023-07-14T18:23:03.395009+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:41.844783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407181+0000","last_change":"2023-07-14T18:23:05.251379+0000","last_active":"2023-07-14T18:24:07.407181+0000","last_peered":"2023-07-14T18:24:07.407181+0000","last_clean":"2023-07-14T18:24:07.407181+0000","last_became_active":"2023-07-14T18:23:05.251082+0000","last_became_peered":"2023-07-14T18:23:05.251082+0000","last_unstale":"2023-07-14T18:24:07.407181+0000","last_undegraded":"2023-07-14T18:24:07.407181+0000","last_fullsized":"2023-07-14T18:24:07.407181+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:34:35.248238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405025+0000","last_change":"2023-07-14T18:23:05.252260+0000","last_active":"2023-07-14T18:24:07.405025+0000","last_peered":"2023-07-14T18:24:07.405025+0000","last_clean":"2023-07-14T18:24:07.405025+0000","last_became_active":"2023-07-14T18:23:05.251967+0000","last_became_peered":"2023-07-14T18:23:05.251967+0000","last_unstale":"2023-07-14T18:24:07.405025+0000","last_undegraded":"2023-07-14T18:24:07.405025+0000","last_fullsized":"2023-07-14T18:24:07.405025+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:25:53.390475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406959+0000","last_change":"2023-07-14T18:23:05.254693+0000","last_active":"2023-07-14T18:24:07.406959+0000","last_peered":"2023-07-14T18:24:07.406959+0000","last_clean":"2023-07-14T18:24:07.406959+0000","last_became_active":"2023-07-14T18:23:05.254395+0000","last_became_peered":"2023-07-14T18:23:05.254395+0000","last_unstale":"2023-07-14T18:24:07.406959+0000","last_undegraded":"2023-07-14T18:24:07.406959+0000","last_fullsized":"2023-07-14T18:24:07.406959+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:40:26.096564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406915+0000","last_change":"2023-07-14T18:23:05.251028+0000","last_active":"2023-07-14T18:24:07.406915+0000","last_peered":"2023-07-14T18:24:07.406915+0000","last_clean":"2023-07-14T18:24:07.406915+0000","last_became_active":"2023-07-14T18:23:05.250720+0000","last_became_peered":"2023-07-14T18:23:05.250720+0000","last_unstale":"2023-07-14T18:24:07.406915+0000","last_undegraded":"2023-07-14T18:24:07.406915+0000","last_fullsized":"2023-07-14T18:24:07.406915+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:32:24.873296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.406046+0000","last_change":"2023-07-14T18:23:05.255123+0000","last_active":"2023-07-14T18:24:07.406046+0000","last_peered":"2023-07-14T18:24:07.406046+0000","last_clean":"2023-07-14T18:24:07.406046+0000","last_became_active":"2023-07-14T18:23:05.254763+0000","last_became_peered":"2023-07-14T18:23:05.254763+0000","last_unstale":"2023-07-14T18:24:07.406046+0000","last_undegraded":"2023-07-14T18:24:07.406046+0000","last_fullsized":"2023-07-14T18:24:07.406046+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:53:58.884740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820615+0000","last_change":"2023-07-14T18:23:05.256086+0000","last_active":"2023-07-14T18:24:07.820615+0000","last_peered":"2023-07-14T18:24:07.820615+0000","last_clean":"2023-07-14T18:24:07.820615+0000","last_became_active":"2023-07-14T18:23:05.255805+0000","last_became_peered":"2023-07-14T18:23:05.255805+0000","last_unstale":"2023-07-14T18:24:07.820615+0000","last_undegraded":"2023-07-14T18:24:07.820615+0000","last_fullsized":"2023-07-14T18:24:07.820615+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T18:23:26.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405289+0000","last_change":"2023-07-14T18:23:05.245449+0000","last_active":"2023-07-14T18:24:07.405289+0000","last_peered":"2023-07-14T18:24:07.405289+0000","last_clean":"2023-07-14T18:24:07.405289+0000","last_became_active":"2023-07-14T18:23:05.244825+0000","last_became_peered":"2023-07-14T18:23:05.244825+0000","last_unstale":"2023-07-14T18:24:07.405289+0000","last_undegraded":"2023-07-14T18:24:07.405289+0000","last_fullsized":"2023-07-14T18:24:07.405289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:02.717582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.407277+0000","last_change":"2023-07-14T18:23:05.247723+0000","last_active":"2023-07-14T18:24:07.407277+0000","last_peered":"2023-07-14T18:24:07.407277+0000","last_clean":"2023-07-14T18:24:07.407277+0000","last_became_active":"2023-07-14T18:23:05.247450+0000","last_became_peered":"2023-07-14T18:23:05.247450+0000","last_unstale":"2023-07-14T18:24:07.407277+0000","last_undegraded":"2023-07-14T18:24:07.407277+0000","last_fullsized":"2023-07-14T18:24:07.407277+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:15:50.630547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598631+0000","last_change":"2023-07-14T18:23:05.254489+0000","last_active":"2023-07-14T18:24:07.598631+0000","last_peered":"2023-07-14T18:24:07.598631+0000","last_clean":"2023-07-14T18:24:07.598631+0000","last_became_active":"2023-07-14T18:23:05.254140+0000","last_became_peered":"2023-07-14T18:23:05.254140+0000","last_unstale":"2023-07-14T18:24:07.598631+0000","last_undegraded":"2023-07-14T18:24:07.598631+0000","last_fullsized":"2023-07-14T18:24:07.598631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:35:06.982685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.410189+0000","last_change":"2023-07-14T18:23:05.241535+0000","last_active":"2023-07-14T18:24:07.410189+0000","last_peered":"2023-07-14T18:24:07.410189+0000","last_clean":"2023-07-14T18:24:07.410189+0000","last_became_active":"2023-07-14T18:23:05.241222+0000","last_became_peered":"2023-07-14T18:23:05.241222+0000","last_unstale":"2023-07-14T18:24:07.410189+0000","last_undegraded":"2023-07-14T18:24:07.410189+0000","last_fullsized":"2023-07-14T18:24:07.410189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:49:31.338811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.598143+0000","last_change":"2023-07-14T18:23:05.249315+0000","last_active":"2023-07-14T18:24:07.598143+0000","last_peered":"2023-07-14T18:24:07.598143+0000","last_clean":"2023-07-14T18:24:07.598143+0000","last_became_active":"2023-07-14T18:23:05.248998+0000","last_became_peered":"2023-07-14T18:23:05.248998+0000","last_unstale":"2023-07-14T18:24:07.598143+0000","last_undegraded":"2023-07-14T18:24:07.598143+0000","last_fullsized":"2023-07-14T18:24:07.598143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:33:21.664881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":22,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.408224+0000","last_change":"2023-07-14T18:23:05.258009+0000","last_active":"2023-07-14T18:24:07.408224+0000","last_peered":"2023-07-14T18:24:07.408224+0000","last_clean":"2023-07-14T18:24:07.408224+0000","last_became_active":"2023-07-14T18:23:05.257674+0000","last_became_peered":"2023-07-14T18:23:05.257674+0000","last_unstale":"2023-07-14T18:24:07.408224+0000","last_undegraded":"2023-07-14T18:24:07.408224+0000","last_fullsized":"2023-07-14T18:24:07.408224+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.457179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.820178+0000","last_change":"2023-07-14T18:23:05.248737+0000","last_active":"2023-07-14T18:24:07.820178+0000","last_peered":"2023-07-14T18:24:07.820178+0000","last_clean":"2023-07-14T18:24:07.820178+0000","last_became_active":"2023-07-14T18:23:05.248388+0000","last_became_peered":"2023-07-14T18:23:05.248388+0000","last_unstale":"2023-07-14T18:24:07.820178+0000","last_undegraded":"2023-07-14T18:24:07.820178+0000","last_fullsized":"2023-07-14T18:24:07.820178+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:16:24.150124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T18:24:07.405223+0000","last_change":"2023-07-14T18:23:05.255318+0000","last_active":"2023-07-14T18:24:07.405223+0000","last_peered":"2023-07-14T18:24:07.405223+0000","last_clean":"2023-07-14T18:24:07.405223+0000","last_became_active":"2023-07-14T18:23:05.255025+0000","last_became_peered":"2023-07-14T18:23:05.255025+0000","last_unstale":"2023-07-14T18:24:07.405223+0000","last_undegraded":"2023-07-14T18:24:07.405223+0000","last_fullsized":"2023-07-14T18:24:07.405223+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:23:04.398773+0000","last_clean_scrub_stamp":"2023-07-14T18:23:04.398773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:22.497297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65561,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":535,"num_read_kb":336,"num_write":234,"num_write_kb":132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":196683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514522,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":477638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.439},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.751},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.80600000000000005}]},{"osd":2,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.495},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.51600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":4,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.63}]},{"osd":5,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.778,"5min":1.778,"15min":1.778},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47399999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.8440000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":0.64800000000000002}]},{"osd":7,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.73399999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.68000000000000005}]},{"osd":9,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.8939999999999999,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"last":0.55200000000000005}]},{"osd":10,"last update":"Fri Jul 14 18:23:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.9510000000000001,"5min":1.9510000000000001,"15min":1.9510000000000001},"last":0.79300000000000004}]}]},{"osd":10,"up_from":69,"seq":296352743454,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28656,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716496,"statfs":{"total":95995035648,"available":95965691904,"internally_reserved":0,"allocated":1671168,"data_stored":1122615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.38300000000000001}]},{"osd":1,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47199999999999998,"15min":0.47099999999999997},"min":{"1min":0.34799999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73499999999999999,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48099999999999998,"15min":0.47899999999999998},"min":{"1min":0.36299999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63,"15min":0.63},"min":{"1min":0.48399999999999999,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.78600000000000003,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.43099999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.80700000000000005,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.51000000000000001}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.59799999999999998,"15min":0.60999999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.65900000000000003,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.627,"15min":0.63300000000000001},"min":{"1min":0.433,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73399999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.41399999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.754,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57999999999999996},"min":{"1min":0.34899999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71199999999999997,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63200000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.58699999999999997,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.82999999999999996,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.72199999999999998,"15min":0.72299999999999998},"min":{"1min":0.59299999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.872,"5min":1.052,"15min":1.052},"last":0.57099999999999995}]},{"osd":6,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.58099999999999996,"5min":0.438,"15min":0.438},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.73599999999999999,"15min":0.73299999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.92800000000000005,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.78800000000000003,"15min":0.78600000000000003},"min":{"1min":0.68300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.92600000000000005,"5min":1.2729999999999999,"15min":1.2729999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.79600000000000004,"15min":0.80200000000000005},"min":{"1min":0.64600000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.93799999999999994,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":9,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.78500000000000003,"15min":0.78600000000000003},"min":{"1min":0.57599999999999996,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.87,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.79200000000000004,"15min":0.80700000000000005},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84699999999999998,"5min":1.256,"15min":1.256},"last":0.754}]},{"osd":11,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.622}]}]},{"osd":9,"up_from":64,"seq":274877906979,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1007616,"data_stored":481322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.499,"15min":0.46400000000000002},"min":{"1min":0.311,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":5.8559999999999999,"5min":5.8559999999999999,"15min":5.8559999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.497,"15min":0.497},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.437}]},{"osd":1,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.68400000000000005,"15min":0.68700000000000006},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":1.76,"5min":1.76,"15min":1.76},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.874,"5min":0.69499999999999995,"15min":0.66500000000000004},"min":{"1min":0.46800000000000003,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":5.5140000000000002,"5min":5.5140000000000002,"15min":5.5140000000000002},"last":0.74199999999999999}]},{"osd":2,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.64200000000000002,"15min":0.64000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.788,"5min":1.788,"15min":1.788},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.71299999999999997,"15min":0.68799999999999994},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":5.641,"5min":5.641,"15min":5.641},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.57199999999999995,"15min":0.57799999999999996},"min":{"1min":0.33800000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.7170000000000001,"5min":1.7170000000000001,"15min":1.7170000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.57899999999999996,"15min":0.58099999999999996},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.9339999999999999,"5min":1.9339999999999999,"15min":1.9339999999999999},"last":0.39400000000000002}]},{"osd":4,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":5.5679999999999996,"5min":5.5679999999999996,"15min":5.5679999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.71199999999999997,"15min":0.71799999999999997},"min":{"1min":0.46200000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.71899999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.78200000000000003,"15min":0.79400000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.6060000000000001,"5min":1.6060000000000001,"15min":1.6060000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.78300000000000003,"15min":0.79300000000000004},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.57899999999999996}]},{"osd":6,"last update":"Fri Jul 14 18:22:42 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.80400000000000005,"15min":0.81399999999999995},"min":{"1min":0.53000000000000003,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.83199999999999996,"15min":0.84999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.73099999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:43 2023","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.85299999999999998,"15min":0.82899999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.79300000000000004,"5min":0.86299999999999999,"15min":0.875},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.60699999999999998}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.79700000000000004,"15min":0.80600000000000005},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.78900000000000003,"15min":0.79700000000000004},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.849,"5min":1.849,"15min":1.849},"last":0.64600000000000002}]},{"osd":11,"last update":"Fri Jul 14 18:24:10 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6339999999999999,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.751},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.9059999999999999,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.621}]}]},{"osd":8,"up_from":57,"seq":244813135912,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":999424,"data_stored":477604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53600000000000003,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.63800000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.54200000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70499999999999996,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.54400000000000004}]},{"osd":1,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59699999999999998,"15min":0.59299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.63600000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59499999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":3.1280000000000001,"5min":3.1280000000000001,"15min":3.1280000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.994,"5min":1.994,"15min":1.994},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.42699999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80100000000000005,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.65700000000000003}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53600000000000003,"15min":0.53700000000000003},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82399999999999995,"5min":0.95099999999999996,"15min":0.95099999999999996},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55800000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.86099999999999999,"5min":0.998,"15min":0.998},"last":0.498}]},{"osd":6,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.65300000000000002,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84299999999999997,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.70199999999999996,"15min":0.70899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.878,"5min":1.026,"15min":1.026},"last":0.84699999999999998}]},{"osd":7,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.68400000000000005,"15min":0.67700000000000005},"min":{"1min":0.47199999999999998,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94499999999999995,"5min":1.073,"15min":1.073},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.96599999999999997,"5min":1.2989999999999999,"15min":1.2989999999999999},"last":0.75}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.70399999999999996,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.679,"5min":1.679,"15min":1.679},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.71099999999999997,"15min":0.69899999999999995},"min":{"1min":0.55200000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.534,"5min":1.534,"15min":1.534},"last":0.90600000000000003}]},{"osd":10,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.74299999999999999,"15min":0.73299999999999998},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.74199999999999999,"15min":0.71999999999999997},"min":{"1min":0.59199999999999997,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.874}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.117,"5min":1.117,"15min":1.117},"last":0.83299999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364843,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28580,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716572,"statfs":{"total":95995035648,"available":95965769728,"internally_reserved":0,"allocated":1593344,"data_stored":1073683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.316,"15min":0.316},"max":{"1min":0.72699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.76800000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.59699999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41199999999999998,"15min":0.39800000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.439,"15min":0.42899999999999999},"min":{"1min":0.28299999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.47099999999999997}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.46000000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.46400000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77600000000000002,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.748,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.86899999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.755}]},{"osd":4,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52900000000000003,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.71999999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51000000000000001,"15min":0.5},"min":{"1min":0.38200000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.67600000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.42999999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.63300000000000001},"min":{"1min":0.54400000000000004,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.89600000000000002,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.62,"15min":0.61599999999999999},"min":{"1min":0.46000000000000002,"5min":0.44,"15min":0.44},"max":{"1min":0.86199999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55900000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.68300000000000005,"15min":0.67100000000000004},"min":{"1min":0.50700000000000001,"5min":0.437,"15min":0.437},"max":{"1min":0.92500000000000004,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.66800000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.90500000000000003,"5min":1.02,"15min":1.02},"last":0.59799999999999998}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.69299999999999995,"15min":0.69099999999999995},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.89500000000000002,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.65800000000000003,"15min":0.65100000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.78300000000000003}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.68999999999999995,"15min":0.68799999999999994},"min":{"1min":0.52800000000000002,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75,"15min":0.748},"min":{"1min":0.57199999999999995,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.97099999999999997,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:24:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.69999999999999996,"15min":0.69599999999999995},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.86199999999999999,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.74199999999999999,"15min":0.73599999999999999},"min":{"1min":0.66300000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.876,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86099999999999999}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81499999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528368,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":472409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38900000000000001,"15min":0.38600000000000001},"min":{"1min":0.26800000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.40200000000000002,"15min":0.40500000000000003},"min":{"1min":0.28299999999999997,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.60999999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.502}]},{"osd":1,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.58199999999999996,"15min":0.56799999999999995},"min":{"1min":0.35699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.78400000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.66700000000000004}]},{"osd":2,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.58599999999999997,"15min":0.57599999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.77600000000000002}]},{"osd":3,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.45700000000000002,"15min":0.45900000000000002},"min":{"1min":0.30599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.497,"15min":0.502},"min":{"1min":0.34499999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71599999999999997,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.754}]},{"osd":4,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60299999999999998,"15min":0.59199999999999997},"min":{"1min":0.46600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93300000000000005,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.63600000000000001},"min":{"1min":0.45800000000000002,"5min":0.436,"15min":0.436},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.58899999999999997}]},{"osd":5,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.64900000000000002,"15min":0.66300000000000003},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96399999999999997,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.66500000000000004,"15min":0.66600000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.95099999999999996,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.79800000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65000000000000002,"15min":0.64300000000000002},"min":{"1min":0.46300000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.68100000000000005,"15min":0.67400000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.80100000000000005,"5min":1.099,"15min":1.099},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.73499999999999999,"15min":0.745},"min":{"1min":0.44500000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.69699999999999995,"15min":0.70799999999999996},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.90100000000000002}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.42199999999999999,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1120000000000001,"5min":1.1120000000000001,"15min":1.1120000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.76000000000000001,"15min":0.76500000000000001},"min":{"1min":0.63,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.79000000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0920000000000001,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.94599999999999995}]},{"osd":11,"last update":"Fri Jul 14 18:24:11 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.91400000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":532263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.38700000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52800000000000002,"15min":0.51100000000000001},"min":{"1min":0.42299999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.78500000000000003}]},{"osd":2,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.65900000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.56899999999999995,"15min":0.53700000000000003},"min":{"1min":0.48899999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.96399999999999997}]},{"osd":3,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.56799999999999995,"15min":0.53400000000000003},"min":{"1min":0.433,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.86699999999999999,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56100000000000005},"min":{"1min":0.39700000000000002,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59099999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.46400000000000002,"15min":0.42999999999999999},"min":{"1min":0.35799999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.71199999999999997,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.48499999999999999,"15min":0.46200000000000002},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.76700000000000002,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.75}]},{"osd":5,"last update":"Fri Jul 14 18:23:19 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82399999999999995,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.63800000000000001,"15min":0.61699999999999999},"min":{"1min":0.55800000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.93600000000000005}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.64000000000000001,"15min":0.61299999999999999},"min":{"1min":0.48499999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.93999999999999995,"5min":1.089,"15min":1.089},"last":0.875},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.64000000000000001,"15min":0.61499999999999999},"min":{"1min":0.51000000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.878,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67000000000000004}]},{"osd":7,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.498,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.0249999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.623,"15min":0.58199999999999996},"min":{"1min":0.40200000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88900000000000001,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.84199999999999997}]},{"osd":8,"last update":"Fri Jul 14 18:23:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.69499999999999995,"15min":0.67900000000000005},"min":{"1min":0.54600000000000004,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.012,"5min":1.1040000000000001,"15min":1.1040000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.66600000000000004,"15min":0.65600000000000003},"min":{"1min":0.52600000000000002,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.73299999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:23:47 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.70099999999999996,"15min":0.69599999999999995},"min":{"1min":0.51200000000000001,"5min":0.5,"15min":0.5},"max":{"1min":0.92000000000000004,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.79600000000000004,"15min":0.78800000000000003},"min":{"1min":0.64000000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.999,"5min":1.173,"15min":1.173},"last":0.98999999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.097,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":1.014},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.81399999999999995,"15min":0.81599999999999995},"min":{"1min":0.59099999999999997,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1,"5min":1,"15min":1},"last":0.92400000000000004}]},{"osd":11,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.131,"5min":1.131,"15min":1.131},"last":1.036}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":999424,"data_stored":474255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.56899999999999995,"15min":0.505},"min":{"1min":0.38800000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.82299999999999995,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.62},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.56000000000000005,"15min":0.51500000000000001},"min":{"1min":0.39400000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.81899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.56699999999999995,"15min":0.55900000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.81100000000000005,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.57199999999999995,"15min":0.57699999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.96599999999999997,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.58399999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.55700000000000005}]},{"osd":4,"last update":"Fri Jul 14 18:24:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.56299999999999994,"15min":0.54700000000000004},"min":{"1min":0.41299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.61499999999999999,"15min":0.57099999999999995},"min":{"1min":0.498,"5min":0.307,"15min":0.307},"max":{"1min":0.82999999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.67700000000000005}]},{"osd":5,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.57299999999999995,"15min":0.55100000000000005},"min":{"1min":0.48599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.89800000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.53100000000000003}]},{"osd":6,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.53100000000000003,"15min":0.51100000000000001},"min":{"1min":0.38,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73599999999999999,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.65500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.70599999999999996}]},{"osd":7,"last update":"Fri Jul 14 18:24:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":1.0760000000000001}]},{"osd":8,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.72899999999999998,"15min":0.73499999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.97399999999999998,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.747,"15min":0.75800000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.0129999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.751}]},{"osd":9,"last update":"Fri Jul 14 18:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63700000000000001,"15min":0.65100000000000002},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.74299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.77300000000000002,"15min":0.78500000000000003},"min":{"1min":0.503,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.85499999999999998,"5min":1.036,"15min":1.036},"last":1.0269999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70299999999999996,"15min":0.70799999999999996},"min":{"1min":0.54500000000000004,"5min":0.442,"15min":0.442},"max":{"1min":0.879,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.80600000000000005,"15min":0.80800000000000005},"min":{"1min":0.65500000000000003,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91400000000000003,"5min":1.075,"15min":1.075},"last":1.23}]},{"osd":11,"last update":"Fri Jul 14 18:23:25 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":1.1890000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.79600000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":533215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49199999999999999,"15min":0.46200000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.51200000000000001,"15min":0.48799999999999999},"min":{"1min":0.39200000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Jul 14 18:23:18 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.34799999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.55700000000000005,"15min":0.52400000000000002},"min":{"1min":0.441,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.624}]},{"osd":3,"last update":"Fri Jul 14 18:23:38 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.56999999999999995,"15min":0.56499999999999995},"min":{"1min":0.46000000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.746},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.55600000000000005,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.751,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58699999999999997}]},{"osd":4,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.63700000000000001}]},{"osd":5,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.47799999999999998,"15min":0.437},"min":{"1min":0.41499999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.35499999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.623,"15min":0.60499999999999998},"min":{"1min":0.54100000000000004,"5min":0.371,"15min":0.371},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.61299999999999999},"min":{"1min":0.376,"5min":0.374,"15min":0.374},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 18:24:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.028,"5min":1.028,"15min":1.028},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77100000000000002}]},{"osd":8,"last update":"Fri Jul 14 18:23:26 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.63100000000000001,"15min":0.61199999999999999},"min":{"1min":0.42699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.66000000000000003,"15min":0.63300000000000001},"min":{"1min":0.45400000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":2.633,"5min":2.633,"15min":2.633},"last":0.67200000000000004}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.72299999999999998,"15min":0.70499999999999996},"min":{"1min":0.55200000000000005,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.75800000000000001,"15min":0.74199999999999999},"min":{"1min":0.56399999999999995,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.81799999999999995}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.78000000000000003,"15min":0.76600000000000001},"min":{"1min":0.59099999999999997,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.621,"15min":1.621},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.86599999999999999,"5min":0.79900000000000004,"15min":0.78800000000000003},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.83999999999999997}]},{"osd":11,"last update":"Fri Jul 14 18:24:03 2023","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.171,"5min":1.171,"15min":1.171},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.70199999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":472157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.41199999999999998,"15min":0.374},"min":{"1min":0.29099999999999998,"5min":0.161,"15min":0.161},"max":{"1min":3.7320000000000002,"5min":3.7320000000000002,"15min":3.7320000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.82899999999999996,"5min":0.47799999999999998,"15min":0.38800000000000001},"min":{"1min":0.27900000000000003,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":8.3140000000000001,"5min":8.3140000000000001,"15min":8.3140000000000001},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.59799999999999998,"15min":0.56799999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":3.2120000000000002,"5min":3.2120000000000002,"15min":3.2120000000000002},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.629,"15min":0.57899999999999996},"min":{"1min":0.41799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"last":0.54100000000000004}]},{"osd":2,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.67600000000000005,"15min":0.58399999999999996},"min":{"1min":0.44700000000000001,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":8.1820000000000004,"5min":8.1820000000000004,"15min":8.1820000000000004},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.60399999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":1.7729999999999999},"last":0.498}]},{"osd":4,"last update":"Fri Jul 14 18:24:02 2023","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":0.68200000000000005,"15min":0.59599999999999997},"min":{"1min":0.45700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":8.2210000000000001,"5min":8.2210000000000001,"15min":8.2210000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.61699999999999999,"15min":0.58499999999999996},"min":{"1min":0.41499999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.67400000000000004}]},{"osd":5,"last update":"Fri Jul 14 18:23:22 2023","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.64400000000000002,"15min":0.58999999999999997},"min":{"1min":0.47399999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":5.8860000000000001,"5min":5.8860000000000001,"15min":5.8860000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.03,"5min":0.71499999999999997,"15min":0.63600000000000001},"min":{"1min":0.495,"5min":0.374,"15min":0.374},"max":{"1min":8.3930000000000007,"5min":8.3930000000000007,"15min":8.3930000000000007},"last":0.69799999999999995}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.68500000000000005,"15min":0.61599999999999999},"min":{"1min":0.38900000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":8.3520000000000003,"5min":8.3520000000000003,"15min":8.3520000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.1919999999999999,"5min":0.69199999999999995,"15min":0.59499999999999997},"min":{"1min":0.34399999999999997,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":12.239000000000001,"5min":12.239000000000001,"15min":12.239000000000001},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.81000000000000005,"15min":0.73399999999999999},"min":{"1min":0.54900000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":11.609999999999999,"5min":11.609999999999999,"15min":11.609999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.125,"5min":0.84499999999999997,"15min":0.79900000000000004},"min":{"1min":0.497,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":9.0559999999999992,"5min":9.0559999999999992,"15min":9.0559999999999992},"last":0.73499999999999999}]},{"osd":8,"last update":"Fri Jul 14 18:22:22 2023","interfaces":[{"interface":"back","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.024,"5min":1.024,"15min":1.024},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.622}]},{"osd":9,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":3.2530000000000001,"5min":3.2530000000000001,"15min":3.2530000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.28,"5min":0.879,"15min":0.81200000000000006},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":11.455,"5min":11.455,"15min":11.455},"last":0.64500000000000002}]},{"osd":10,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":0.91000000000000003,"15min":0.86799999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":9.0150000000000006,"5min":9.0150000000000006,"15min":9.0150000000000006},"last":0.75700000000000001},{"interface":"front","average":{"1min":1.304,"5min":0.90400000000000003,"15min":0.83799999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":12.291,"5min":12.291,"15min":12.291},"last":0.747}]},{"osd":11,"last update":"Fri Jul 14 18:24:08 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.26,"15min":1.26},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":11.263999999999999,"5min":11.263999999999999,"15min":11.263999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.77700000000000002}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":476483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:02 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48799999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.48499999999999999},"min":{"1min":0.374,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72399999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.374}]},{"osd":1,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.39200000000000002,"15min":0.37},"min":{"1min":0.29999999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54400000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.41899999999999998,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.621,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.442}]},{"osd":2,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.628,"15min":0.58099999999999996},"min":{"1min":0.57499999999999996,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.1909999999999998,"5min":3.1909999999999998,"15min":3.1909999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.56200000000000006},"min":{"1min":0.495,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.73099999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.57699999999999996}]},{"osd":3,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56100000000000005,"15min":0.53000000000000003},"min":{"1min":0.35499999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78300000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57299999999999995,"15min":0.53700000000000003},"min":{"1min":0.40400000000000003,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":2.3210000000000002,"5min":2.3210000000000002,"15min":2.3210000000000002},"last":0.86299999999999999}]},{"osd":5,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.59999999999999998,"15min":0.58299999999999996},"min":{"1min":0.40999999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.873,"5min":0.90200000000000002,"15min":0.90200000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.63,"15min":0.61299999999999999},"min":{"1min":0.50900000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Jul 14 18:23:44 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.436,"5min":0.374,"15min":0.374},"max":{"1min":0.78000000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.62,"5min":0.65800000000000003,"15min":0.65300000000000002},"min":{"1min":0.47499999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89000000000000001,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67700000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59099999999999997,"15min":0.60499999999999998},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.86399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60199999999999998,"15min":0.60499999999999998},"min":{"1min":0.44700000000000001,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.75,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.65900000000000003}]},{"osd":8,"last update":"Fri Jul 14 18:23:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.70199999999999996,"15min":0.69999999999999996},"min":{"1min":0.46700000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.90600000000000003,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.67900000000000005,"15min":0.67000000000000004},"min":{"1min":0.496,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.88,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.71599999999999997,"15min":0.71199999999999997},"min":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.754,"5min":1.754,"15min":1.754},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73499999999999999,"15min":0.74099999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.74199999999999999}]},{"osd":10,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.74199999999999999,"15min":0.75},"min":{"1min":0.49399999999999999,"5min":0.378,"15min":0.378},"max":{"1min":0.98099999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.72099999999999997,"15min":0.72499999999999998},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.77100000000000002}]},{"osd":11,"last update":"Fri Jul 14 18:24:06 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.81899999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28588,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716564,"statfs":{"total":95995035648,"available":95965761536,"internally_reserved":0,"allocated":1601536,"data_stored":1069303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53100000000000003,"15min":0.51300000000000001},"min":{"1min":0.42999999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68500000000000005,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.624},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53800000000000003,"15min":0.52700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.81599999999999995,"5min":0.873,"15min":0.873},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.45500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55700000000000005,"15min":0.54300000000000004},"min":{"1min":0.47699999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.66800000000000004}]},{"osd":2,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.40999999999999998,"15min":0.39100000000000001},"min":{"1min":0.34000000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.46600000000000003}]},{"osd":3,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.42999999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.175,"5min":1.175,"15min":1.175},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.58299999999999996,"15min":0.55500000000000005},"min":{"1min":0.495,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.69399999999999995}]},{"osd":4,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.59399999999999997,"15min":0.57899999999999996},"min":{"1min":0.41899999999999998,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.41199999999999998,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.71699999999999997}]},{"osd":6,"last update":"Fri Jul 14 18:23:42 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.621,"15min":0.60499999999999998},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78500000000000003,"5min":1.1060000000000001,"15min":1.1060000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.65100000000000002,"15min":0.63700000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.83399999999999996,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.56100000000000005}]},{"osd":7,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.68000000000000005,"15min":0.66000000000000003},"min":{"1min":0.48699999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.94499999999999995,"5min":1.075,"15min":1.075},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.69299999999999995,"15min":0.67900000000000005},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.84699999999999998,"5min":1.0289999999999999,"15min":1.0289999999999999},"last":0.83299999999999996}]},{"osd":8,"last update":"Fri Jul 14 18:23:23 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.622,"15min":0.60999999999999999},"min":{"1min":0.41199999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.61399999999999999,"15min":0.60399999999999998},"min":{"1min":0.45500000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60599999999999998}]},{"osd":9,"last update":"Fri Jul 14 18:23:46 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.77700000000000002,"15min":0.78300000000000003},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.89600000000000002,"5min":1.087,"15min":1.087},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.79700000000000004,"15min":0.80200000000000005},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":0.98799999999999999,"5min":1.206,"15min":1.206},"last":0.79300000000000004}]},{"osd":10,"last update":"Fri Jul 14 18:24:07 2023","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.81200000000000006,"15min":0.81299999999999994},"min":{"1min":0.55100000000000005,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":0.95699999999999996,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.77400000000000002,"15min":0.77100000000000002},"min":{"1min":0.53700000000000003,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.873,"5min":1.175,"15min":1.175},"last":0.87}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":60239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T18:24:23.163 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T18:24:23.163 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T18:24:23.163 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T18:24:23.164 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T18:24:23.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:23 smithi064 ceph-mon[96720]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:23.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:23 smithi144 ceph-mon[98417]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:23.726 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:23.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:23 smithi105 ceph-mon[98013]: pgmap v295: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:24.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:24 smithi064 ceph-mon[96720]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:24.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:24 smithi144 ceph-mon[98417]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:24.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:24 smithi105 ceph-mon[98013]: from='client.15129 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:24:24.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 -- 172.21.15.64:0/3641380363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d80fee90 msgr2=0x7f72d80ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 --2- 172.21.15.64:0/3641380363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d80fee90 0x7f72d80ff260 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f72c00099c0 tx=0x7f72c0009cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 -- 172.21.15.64:0/3641380363 shutdown_connections 2023-07-14T18:24:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 --2- 172.21.15.64:0/3641380363 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d8103f30 0x7f72d810bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 --2- 172.21.15.64:0/3641380363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d80fee90 0x7f72d80ff260 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 --2- 172.21.15.64:0/3641380363 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8103800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 -- 172.21.15.64:0/3641380363 >> 172.21.15.64:0/3641380363 conn(0x7f72d8075ae0 msgr2=0x7f72d8075ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:24.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 -- 172.21.15.64:0/3641380363 shutdown_connections 2023-07-14T18:24:24.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.866+0000 7f72dd310700 1 -- 172.21.15.64:0/3641380363 wait complete. 2023-07-14T18:24:24.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 Processor -- start 2023-07-14T18:24:24.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 -- start start 2023-07-14T18:24:24.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d80fee90 0x7f72d8196b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:24.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8197060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:24.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:24.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d8197970 con 0x7f72d8103f30 2023-07-14T18:24:24.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81a20e0 con 0x7f72d80ff830 2023-07-14T18:24:24.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72dd310700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d81a2250 con 0x7f72d80fee90 2023-07-14T18:24:24.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:24.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47964/0 (socket says 172.21.15.64:47964) 2023-07-14T18:24:24.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 -- 172.21.15.64:0/4001253446 learned_addr learned my addr 172.21.15.64:0/4001253446 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d67fc700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8197060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d6ffd700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d80fee90 0x7f72d8196b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 -- 172.21.15.64:0/4001253446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d80fee90 msgr2=0x7f72d8196b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d80fee90 0x7f72d8196b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 -- 172.21.15.64:0/4001253446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 msgr2=0x7f72d8197060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8197060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:24.927 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d77fe700 1 -- 172.21.15.64:0/4001253446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72c00095f0 con 0x7f72d8103f30 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.867+0000 7f72d67fc700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72d77fe700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f72d0007ee0 tx=0x7f72d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0017070 con 0x7f72d8103f30 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d81a2530 con 0x7f72d8103f30 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f72d000f040 con 0x7f72d8103f30 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d0005710 con 0x7f72d8103f30 2023-07-14T18:24:24.928 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.868+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d81a29e0 con 0x7f72d8103f30 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.869+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72d804e340 con 0x7f72d8103f30 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.870+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f72d0005870 con 0x7f72d8103f30 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.871+0000 7f72cffff700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 0x7f72c40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.871+0000 7f72d6ffd700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 0x7f72c40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.871+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f72d00cc670 con 0x7f72d8103f30 2023-07-14T18:24:24.929 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.871+0000 7f72d6ffd700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 0x7f72c40a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72c0006010 tx=0x7f72c0003680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:24.930 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:24.875+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f72d00958f0 con 0x7f72d8103f30 2023-07-14T18:24:25.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.118+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f72d81a2c00 con 0x7f72d8103f30 2023-07-14T18:24:25.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.119+0000 7f72cffff700 1 -- 172.21.15.64:0/4001253446 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f72d0098f10 con 0x7f72d8103f30 2023-07-14T18:24:25.176 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:25.177 INFO:teuthology.orchestra.run.smithi064.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T18:24:25.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 msgr2=0x7f72c40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 0x7f72c40a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f72c0006010 tx=0x7f72c0003680 comp rx=0 tx=0).stop 2023-07-14T18:24:25.177 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 msgr2=0x7f72d81a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f72d0007ee0 tx=0x7f72d000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 shutdown_connections 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f72d80fee90 0x7f72d8196b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f72c40a11e0 0x7f72c40a3690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72d8103f30 0x7f72d81a1ba0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 --2- 172.21.15.64:0/4001253446 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f72d80ff830 0x7f72d8197060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.178 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 >> 172.21.15.64:0/4001253446 conn(0x7f72d8075ae0 msgr2=0x7f72d80fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:25.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 shutdown_connections 2023-07-14T18:24:25.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.121+0000 7f72dd310700 1 -- 172.21.15.64:0/4001253446 wait complete. 2023-07-14T18:24:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:25 smithi064 ceph-mon[96720]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:25.565 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:25 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4001253446' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:24:25.581 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T18:24:25.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:25.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:25 smithi144 ceph-mon[98417]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:25.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:25 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4001253446' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:24:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3052157738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0881006b0 msgr2=0x7fd088106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3052157738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0881006b0 0x7fd088106f70 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd08000b0a0 tx=0x7fd08000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3052157738 shutdown_connections 2023-07-14T18:24:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3052157738 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0880feb30 0x7fd0880fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3052157738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0881006b0 0x7fd088106f70 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3052157738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880ffd20 0x7fd088100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3052157738 >> 172.21.15.64:0/3052157738 conn(0x7fd0880fa150 msgr2=0x7fd0880fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3052157738 shutdown_connections 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.659+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3052157738 wait complete. 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 Processor -- start 2023-07-14T18:24:25.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 -- start start 2023-07-14T18:24:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 0x7fd088195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0881006b0 0x7fd08819cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08819d4f0 con 0x7fd0880ffd20 2023-07-14T18:24:25.715 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08819d630 con 0x7fd0880feb30 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08e9b1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd08819d7c0 con 0x7fd0881006b0 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47976/0 (socket says 172.21.15.64:47976) 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 -- 172.21.15.64:0/3650497290 learned_addr learned my addr 172.21.15.64:0/3650497290 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd087fff700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 0x7fd088195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd08cf4e700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0881006b0 0x7fd08819cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:25.716 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 -- 172.21.15.64:0/3650497290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0881006b0 msgr2=0x7fd08819cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0881006b0 0x7fd08819cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 -- 172.21.15.64:0/3650497290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 msgr2=0x7fd088195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 0x7fd088195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 -- 172.21.15.64:0/3650497290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080009d00 con 0x7fd0880ffd20 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd087fff700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 0x7fd088195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.660+0000 7fd0877fe700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e9a0 tx=0x7fd07c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:25.717 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.661+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c009790 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.661+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd07c00cbd0 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.661+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd08819daa0 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.661+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c018680 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.661+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08819e000 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.662+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd08810b3e0 con 0x7fd0880ffd20 2023-07-14T18:24:25.718 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.663+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd07c00cd40 con 0x7fd0880ffd20 2023-07-14T18:24:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.663+0000 7fd0857fa700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 0x7fd0700a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.663+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd07c014070 con 0x7fd0880ffd20 2023-07-14T18:24:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.663+0000 7fd087fff700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 0x7fd0700a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:25.719 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.664+0000 7fd087fff700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 0x7fd0700a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd078005fd0 tx=0x7fd078005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:25.720 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.668+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd07c051780 con 0x7fd0880ffd20 2023-07-14T18:24:25.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:25 smithi105 ceph-mon[98013]: pgmap v296: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:25.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:25 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4001253446' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T18:24:25.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.868+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd08810b5d0 con 0x7fd0880ffd20 2023-07-14T18:24:25.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.870+0000 7fd0857fa700 1 -- 172.21.15.64:0/3650497290 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7fd07c099de0 con 0x7fd0880ffd20 2023-07-14T18:24:25.923 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:25.923 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:25.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 msgr2=0x7fd0700a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 0x7fd0700a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd078005fd0 tx=0x7fd078005f00 comp rx=0 tx=0).stop 2023-07-14T18:24:25.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 msgr2=0x7fd088195eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:25.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd07c00e9a0 tx=0x7fd07c00ed60 comp rx=0 tx=0).stop 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 shutdown_connections 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd0881006b0 0x7fd08819cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd0700a12a0 0x7fd0700a3750 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd0880feb30 0x7fd088195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 --2- 172.21.15.64:0/3650497290 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd0880ffd20 0x7fd088195eb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.872+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 >> 172.21.15.64:0/3650497290 conn(0x7fd0880fa150 msgr2=0x7fd0880fc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.873+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 shutdown_connections 2023-07-14T18:24:25.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:25.873+0000 7fd08e9b1700 1 -- 172.21.15.64:0/3650497290 wait complete. 2023-07-14T18:24:25.938 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:24:25.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:24:26.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 -- 172.21.15.64:0/3820144766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c100750 msgr2=0x7fa73c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 --2- 172.21.15.64:0/3820144766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c100750 0x7fa73c102b30 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa724009990 tx=0x7fa724009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 -- 172.21.15.64:0/3820144766 shutdown_connections 2023-07-14T18:24:26.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 --2- 172.21.15.64:0/3820144766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c105990 0x7fa73c107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 --2- 172.21.15.64:0/3820144766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c100750 0x7fa73c102b30 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 --2- 172.21.15.64:0/3820144766 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c103070 0x7fa73c105450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 -- 172.21.15.64:0/3820144766 >> 172.21.15.64:0/3820144766 conn(0x7fa73c0fa130 msgr2=0x7fa73c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 -- 172.21.15.64:0/3820144766 shutdown_connections 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 -- 172.21.15.64:0/3820144766 wait complete. 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.015+0000 7fa741333700 1 Processor -- start 2023-07-14T18:24:26.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 -- start start 2023-07-14T18:24:26.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c100750 0x7fa73c1958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c103070 0x7fa73c195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c19d450 con 0x7fa73c105990 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c19d590 con 0x7fa73c103070 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa741333700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73c19d720 con 0x7fa73c100750 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73affd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c100750 0x7fa73c1958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:47992/0 (socket says 172.21.15.64:47992) 2023-07-14T18:24:26.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 -- 172.21.15.64:0/3554975916 learned_addr learned my addr 172.21.15.64:0/3554975916 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73a7fc700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c103070 0x7fa73c195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 -- 172.21.15.64:0/3554975916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c100750 msgr2=0x7fa73c1958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c100750 0x7fa73c1958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 -- 172.21.15.64:0/3554975916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c103070 msgr2=0x7fa73c195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c103070 0x7fa73c195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.016+0000 7fa73b7fe700 1 -- 172.21.15.64:0/3554975916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7240095f0 con 0x7fa73c105990 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa73b7fe700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa73400ef30 tx=0x7fa73400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa734018070 con 0x7fa73c105990 2023-07-14T18:24:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73c19d9a0 con 0x7fa73c105990 2023-07-14T18:24:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa73400f040 con 0x7fa73c105990 2023-07-14T18:24:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa73400cbc0 con 0x7fa73c105990 2023-07-14T18:24:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.017+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73c19de60 con 0x7fa73c105990 2023-07-14T18:24:26.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.018+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa73c0fbca0 con 0x7fa73c105990 2023-07-14T18:24:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.019+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa73400cd80 con 0x7fa73c105990 2023-07-14T18:24:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.020+0000 7fa733fff700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 0x7fa7280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.020+0000 7fa73affd700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 0x7fa7280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.020+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa7340cce40 con 0x7fa73c105990 2023-07-14T18:24:26.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.020+0000 7fa73affd700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 0x7fa7280a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa724006010 tx=0x7fa72400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.023+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa7340961b0 con 0x7fa73c105990 2023-07-14T18:24:26.275 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.223+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa73c02ca70 con 0x7fa73c105990 2023-07-14T18:24:26.276 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.223+0000 7fa733fff700 1 -- 172.21.15.64:0/3554975916 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa7340997d0 con 0x7fa73c105990 2023-07-14T18:24:26.276 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:24:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 msgr2=0x7fa7280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.278 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 0x7fa7280a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa724006010 tx=0x7fa72400b260 comp rx=0 tx=0).stop 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 msgr2=0x7fa73c19ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fa73400ef30 tx=0x7fa73400c3a0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 shutdown_connections 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa73c100750 0x7fa73c1958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa7280a11e0 0x7fa7280a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa73c105990 0x7fa73c19ce80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.279 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 --2- 172.21.15.64:0/3554975916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa73c103070 0x7fa73c195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 >> 172.21.15.64:0/3554975916 conn(0x7fa73c0fa130 msgr2=0x7fa73c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 shutdown_connections 2023-07-14T18:24:26.280 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.226+0000 7fa741333700 1 -- 172.21.15.64:0/3554975916 wait complete. 2023-07-14T18:24:26.291 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:24:26.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 -- 172.21.15.64:0/113899338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840ffd30 msgr2=0x7f7984100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 --2- 172.21.15.64:0/113899338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840ffd30 0x7f7984100180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7978009990 tx=0x7f7978009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 -- 172.21.15.64:0/113899338 shutdown_connections 2023-07-14T18:24:26.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 --2- 172.21.15.64:0/113899338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79840feb40 0x7f79840fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 --2- 172.21.15.64:0/113899338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79841006c0 0x7f7984106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 --2- 172.21.15.64:0/113899338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840ffd30 0x7f7984100180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 -- 172.21.15.64:0/113899338 >> 172.21.15.64:0/113899338 conn(0x7f79840fa160 msgr2=0x7f79840fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 -- 172.21.15.64:0/113899338 shutdown_connections 2023-07-14T18:24:26.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.368+0000 7f798906e700 1 -- 172.21.15.64:0/113899338 wait complete. 2023-07-14T18:24:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.369+0000 7f798906e700 1 Processor -- start 2023-07-14T18:24:26.427 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.369+0000 7f798906e700 1 -- start start 2023-07-14T18:24:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840ffd30 0x7f7984101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.428 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79841006c0 0x7f7984101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984101df0 con 0x7f79840feb40 2023-07-14T18:24:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984101f60 con 0x7f79840ffd30 2023-07-14T18:24:26.429 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798419f900 con 0x7f79841006c0 2023-07-14T18:24:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48002/0 (socket says 172.21.15.64:48002) 2023-07-14T18:24:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 -- 172.21.15.64:0/449391849 learned_addr learned my addr 172.21.15.64:0/449391849 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:26.430 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798359e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79841006c0 0x7f7984101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798259c700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840ffd30 0x7f7984101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 -- 172.21.15.64:0/449391849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79841006c0 msgr2=0x7f7984101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79841006c0 0x7f7984101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 -- 172.21.15.64:0/449391849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840ffd30 msgr2=0x7f7984101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840ffd30 0x7f7984101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 -- 172.21.15.64:0/449391849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79780095f0 con 0x7f79840feb40 2023-07-14T18:24:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f7982d9d700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f797400d980 tx=0x7f797400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974004020 con 0x7f79840feb40 2023-07-14T18:24:26.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f797400f690 con 0x7f79840feb40 2023-07-14T18:24:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79740039e0 con 0x7f79840feb40 2023-07-14T18:24:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798419fbe0 con 0x7f79840feb40 2023-07-14T18:24:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.370+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841a00f0 con 0x7f79840feb40 2023-07-14T18:24:26.433 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.371+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798410a210 con 0x7f79840feb40 2023-07-14T18:24:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.373+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f797400f800 con 0x7f79840feb40 2023-07-14T18:24:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.373+0000 7f796bfff700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 0x7f796c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.373+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f79740ccc30 con 0x7f79840feb40 2023-07-14T18:24:26.434 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.373+0000 7f798259c700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 0x7f796c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.374+0000 7f798259c700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 0x7f796c0a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f797800c010 tx=0x7f7978005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.377+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7974095f90 con 0x7f79840feb40 2023-07-14T18:24:26.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3650497290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:26.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:26 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3554975916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:26.628 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.576+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f79841a0700 con 0x7f79840feb40 2023-07-14T18:24:26.629 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.576+0000 7f796bfff700 1 -- 172.21.15.64:0/449391849 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f79740995b0 con 0x7f79840feb40 2023-07-14T18:24:26.629 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:26.630 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 msgr2=0x7f796c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 0x7f796c0a3710 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f797800c010 tx=0x7f7978005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 msgr2=0x7f7984102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f797400d980 tx=0x7f797400dc90 comp rx=0 tx=0).stop 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 shutdown_connections 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79841006c0 0x7f7984101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.631 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f796c0a1260 0x7f796c0a3710 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79840feb40 0x7f7984102ad0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 --2- 172.21.15.64:0/449391849 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840ffd30 0x7f7984101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 >> 172.21.15.64:0/449391849 conn(0x7f79840fa160 msgr2=0x7f7984068720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 shutdown_connections 2023-07-14T18:24:26.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.578+0000 7f798906e700 1 -- 172.21.15.64:0/449391849 wait complete. 2023-07-14T18:24:26.643 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:24:26.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:26 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3650497290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:26.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:26 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3554975916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:26.772 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 -- 172.21.15.64:0/2376675521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac41000d0 msgr2=0x7fdac41004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 --2- 172.21.15.64:0/2376675521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac41000d0 0x7fdac41004e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009990 tx=0x7fdab4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 -- 172.21.15.64:0/2376675521 shutdown_connections 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 --2- 172.21.15.64:0/2376675521 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 0x7fdac4106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 --2- 172.21.15.64:0/2376675521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac41000d0 0x7fdac41004e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 --2- 172.21.15.64:0/2376675521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac40fed20 0x7fdac40ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.773 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 -- 172.21.15.64:0/2376675521 >> 172.21.15.64:0/2376675521 conn(0x7fdac40f98a0 msgr2=0x7fdac40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.774 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.719+0000 7fdaca870700 1 -- 172.21.15.64:0/2376675521 shutdown_connections 2023-07-14T18:24:26.774 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.720+0000 7fdaca870700 1 -- 172.21.15.64:0/2376675521 wait complete. 2023-07-14T18:24:26.774 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.720+0000 7fdaca870700 1 Processor -- start 2023-07-14T18:24:26.774 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.720+0000 7fdaca870700 1 -- start start 2023-07-14T18:24:26.775 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.720+0000 7fdaca870700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.720+0000 7fdaca870700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 0x7fdac419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac41000d0 0x7fdac419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac419d530 con 0x7fdac40fed20 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac419d6a0 con 0x7fdac41000d0 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac419d860 con 0x7fdac40ff6d0 2023-07-14T18:24:26.776 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48016/0 (socket says 172.21.15.64:48016) 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 -- 172.21.15.64:0/1195696161 learned_addr learned my addr 172.21.15.64:0/1195696161 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac8e0d700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac41000d0 0x7fdac419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac37fe700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 0x7fdac419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 -- 172.21.15.64:0/1195696161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 msgr2=0x7fdac419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.777 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 0x7fdac419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 -- 172.21.15.64:0/1195696161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac41000d0 msgr2=0x7fdac419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac41000d0 0x7fdac419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 -- 172.21.15.64:0/1195696161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab40095f0 con 0x7fdac40fed20 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac3fff700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdab4004730 tx=0x7fdab4004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab401d070 con 0x7fdac40fed20 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdab4004cc0 con 0x7fdac40fed20 2023-07-14T18:24:26.778 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac419dae0 con 0x7fdac40fed20 2023-07-14T18:24:26.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdab4005060 con 0x7fdac40fed20 2023-07-14T18:24:26.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.721+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac419dea0 con 0x7fdac40fed20 2023-07-14T18:24:26.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.722+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac4106290 con 0x7fdac40fed20 2023-07-14T18:24:26.779 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.723+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fdab4003b70 con 0x7fdac40fed20 2023-07-14T18:24:26.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.724+0000 7fdac17fa700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 0x7fdaac0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:26.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.724+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdab40ce180 con 0x7fdac40fed20 2023-07-14T18:24:26.780 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.724+0000 7fdac37fe700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 0x7fdaac0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:26.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.724+0000 7fdac37fe700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 0x7fdaac0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdab8005950 tx=0x7fdab80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:26.781 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.727+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdab40974f0 con 0x7fdac40fed20 2023-07-14T18:24:26.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:26 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3650497290' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:26.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:26 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3554975916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:26.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.927+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdac402cb30 con 0x7fdac40fed20 2023-07-14T18:24:26.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.928+0000 7fdac17fa700 1 -- 172.21.15.64:0/1195696161 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdab409ab10 con 0x7fdac40fed20 2023-07-14T18:24:26.981 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:24:26.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 msgr2=0x7fdaac0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 0x7fdaac0a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdab8005950 tx=0x7fdab80058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 msgr2=0x7fdac40fe7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fdab4004730 tx=0x7fdab4004760 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 shutdown_connections 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fdaac0a11e0 0x7fdaac0a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdac41000d0 0x7fdac419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdac40fed20 0x7fdac40fe7a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 --2- 172.21.15.64:0/1195696161 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fdac40ff6d0 0x7fdac419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:26.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 >> 172.21.15.64:0/1195696161 conn(0x7fdac40f98a0 msgr2=0x7fdac41033d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:26.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 shutdown_connections 2023-07-14T18:24:26.984 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:26.930+0000 7fdaca870700 1 -- 172.21.15.64:0/1195696161 wait complete. 2023-07-14T18:24:26.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:24:27.124 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 -- 172.21.15.64:0/1195033203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00feb30 msgr2=0x7fd5b00fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1195033203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00feb30 0x7fd5b00fef40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd5a0009990 tx=0x7fd5a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 -- 172.21.15.64:0/1195033203 shutdown_connections 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1195033203 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b01006b0 0x7fd5b0106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1195033203 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00feb30 0x7fd5b00fef40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1195033203 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b00ffd20 0x7fd5b0100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.125 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 -- 172.21.15.64:0/1195033203 >> 172.21.15.64:0/1195033203 conn(0x7fd5b00fa190 msgr2=0x7fd5b00fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:27.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 -- 172.21.15.64:0/1195033203 shutdown_connections 2023-07-14T18:24:27.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.071+0000 7fd5b5573700 1 -- 172.21.15.64:0/1195033203 wait complete. 2023-07-14T18:24:27.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 Processor -- start 2023-07-14T18:24:27.126 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 -- start start 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b00feb30 0x7fd5b0069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b01006b0 0x7fd5b019cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019d410 con 0x7fd5b00ffd20 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019d580 con 0x7fd5b00feb30 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5b5573700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b019d740 con 0x7fd5b01006b0 2023-07-14T18:24:27.128 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.072+0000 7fd5ae7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48030/0 (socket says 172.21.15.64:48030) 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 -- 172.21.15.64:0/1735388068 learned_addr learned my addr 172.21.15.64:0/1735388068 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5af7fe700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b01006b0 0x7fd5b019cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5aeffd700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b00feb30 0x7fd5b0069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 -- 172.21.15.64:0/1735388068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b01006b0 msgr2=0x7fd5b019cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b01006b0 0x7fd5b019cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.129 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 -- 172.21.15.64:0/1735388068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b00feb30 msgr2=0x7fd5b0069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b00feb30 0x7fd5b0069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 -- 172.21.15.64:0/1735388068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a00095f0 con 0x7fd5b00ffd20 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5ae7fc700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400d950 tx=0x7fd5a400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400f890 con 0x7fd5b00ffd20 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd5a400fe80 con 0x7fd5b00ffd20 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b019da20 con 0x7fd5b00ffd20 2023-07-14T18:24:27.130 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.073+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400e3b0 con 0x7fd5b00ffd20 2023-07-14T18:24:27.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.074+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b019df30 con 0x7fd5b00ffd20 2023-07-14T18:24:27.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.074+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b0105ab0 con 0x7fd5b00ffd20 2023-07-14T18:24:27.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.076+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd5a4010460 con 0x7fd5b00ffd20 2023-07-14T18:24:27.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.077+0000 7fd597fff700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.077+0000 7fd5aeffd700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.077+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd5a40ccb80 con 0x7fd5b00ffd20 2023-07-14T18:24:27.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.077+0000 7fd5aeffd700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 0x7fd5980a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000c010 tx=0x7fd5a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.080+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5a4095e00 con 0x7fd5b00ffd20 2023-07-14T18:24:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.280+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd5b002cb30 con 0x7fd5b00ffd20 2023-07-14T18:24:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.280+0000 7fd597fff700 1 -- 172.21.15.64:0/1735388068 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd5a40160a0 con 0x7fd5b00ffd20 2023-07-14T18:24:27.333 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 msgr2=0x7fd5980a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 0x7fd5980a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000c010 tx=0x7fd5a0005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 msgr2=0x7fd5b019a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400d950 tx=0x7fd5a400dc60 comp rx=0 tx=0).stop 2023-07-14T18:24:27.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 shutdown_connections 2023-07-14T18:24:27.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd5b01006b0 0x7fd5b019cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd5980a11e0 0x7fd5980a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd5b00feb30 0x7fd5b0069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 --2- 172.21.15.64:0/1735388068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5b00ffd20 0x7fd5b019a630 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 >> 172.21.15.64:0/1735388068 conn(0x7fd5b00fa190 msgr2=0x7fd5b00fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:27.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 shutdown_connections 2023-07-14T18:24:27.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.282+0000 7fd5b5573700 1 -- 172.21.15.64:0/1735388068 wait complete. 2023-07-14T18:24:27.347 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 -- 172.21.15.64:0/3600333062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee0100680 msgr2=0x7fcee0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3600333062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee0100680 0x7fcee0106f40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fced800b0a0 tx=0x7fced800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 -- 172.21.15.64:0/3600333062 shutdown_connections 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3600333062 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 0x7fcee00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3600333062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee0100680 0x7fcee0106f40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.475 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3600333062 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee00ffcf0 0x7fcee0100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 -- 172.21.15.64:0/3600333062 >> 172.21.15.64:0/3600333062 conn(0x7fcee00fa100 msgr2=0x7fcee00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:27.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 -- 172.21.15.64:0/3600333062 shutdown_connections 2023-07-14T18:24:27.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.422+0000 7fcee4e64700 1 -- 172.21.15.64:0/3600333062 wait complete. 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 Processor -- start 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 -- start start 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 0x7fcee0068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0100680 0x7fcee019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee019d480 con 0x7fcee00ffcf0 2023-07-14T18:24:27.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee019d5f0 con 0x7fcee0100680 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcee4e64700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee019d7b0 con 0x7fcee00feb00 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48046/0 (socket says 172.21.15.64:48046) 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 -- 172.21.15.64:0/3891732827 learned_addr learned my addr 172.21.15.64:0/3891732827 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fcede59c700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 0x7fcee0068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceded9d700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0100680 0x7fcee019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 -- 172.21.15.64:0/3891732827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 msgr2=0x7fcee0068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 0x7fcee0068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 -- 172.21.15.64:0/3891732827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0100680 msgr2=0x7fcee019cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0100680 0x7fcee019cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.423+0000 7fceddd9b700 1 -- 172.21.15.64:0/3891732827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8009d00 con 0x7fcee00ffcf0 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fceddd9b700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fced400e9a0 tx=0x7fced400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4009790 con 0x7fcee00ffcf0 2023-07-14T18:24:27.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fced400cbd0 con 0x7fcee00ffcf0 2023-07-14T18:24:27.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee019da90 con 0x7fcee00ffcf0 2023-07-14T18:24:27.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced4018680 con 0x7fcee00ffcf0 2023-07-14T18:24:27.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.424+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee019dff0 con 0x7fcee00ffcf0 2023-07-14T18:24:27.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.425+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee0105ff0 con 0x7fcee00ffcf0 2023-07-14T18:24:27.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.426+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fced400cd40 con 0x7fcee00ffcf0 2023-07-14T18:24:27.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.426+0000 7fcecf7fe700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 0x7fcec80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.427+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fced4014070 con 0x7fcee00ffcf0 2023-07-14T18:24:27.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.427+0000 7fcede59c700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 0x7fcec80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.427+0000 7fcede59c700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 0x7fcec80a3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fced0005950 tx=0x7fced00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.431+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fced4051780 con 0x7fcee00ffcf0 2023-07-14T18:24:27.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:27 smithi064 ceph-mon[96720]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:27.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:27 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/449391849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:27 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1195696161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:27 smithi144 ceph-mon[98417]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:27.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:27 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/449391849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:27 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1195696161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.727 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.675+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcee002c8e0 con 0x7fcee00ffcf0 2023-07-14T18:24:27.728 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.676+0000 7fcecf7fe700 1 -- 172.21.15.64:0/3891732827 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7fced4099e70 con 0x7fcee00ffcf0 2023-07-14T18:24:27.728 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:27.728 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:24:27.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 msgr2=0x7fcec80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 0x7fcec80a3750 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fced0005950 tx=0x7fced00058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 msgr2=0x7fcee019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fced400e9a0 tx=0x7fced400ed60 comp rx=0 tx=0).stop 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 shutdown_connections 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fcee00feb00 0x7fcee0068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fcec80a12a0 0x7fcec80a3750 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcee0100680 0x7fcee019cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 --2- 172.21.15.64:0/3891732827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcee00ffcf0 0x7fcee019a6a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 >> 172.21.15.64:0/3891732827 conn(0x7fcee00fa100 msgr2=0x7fcee0103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:27.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 shutdown_connections 2023-07-14T18:24:27.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.678+0000 7fcee4e64700 1 -- 172.21.15.64:0/3891732827 wait complete. 2023-07-14T18:24:27.732 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:27.743 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:27.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:27 smithi105 ceph-mon[98013]: pgmap v297: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:27.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:27 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/449391849' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:27 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1195696161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:27.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 -- 172.21.15.64:0/1356472141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7200feb00 msgr2=0x7fc7200fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1356472141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7200feb00 0x7fc7200fef10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc710009990 tx=0x7fc710009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 -- 172.21.15.64:0/1356472141 shutdown_connections 2023-07-14T18:24:27.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1356472141 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc720100680 0x7fc720106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1356472141 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc7200feb00 0x7fc7200fef10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1356472141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 0x7fc720100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 -- 172.21.15.64:0/1356472141 >> 172.21.15.64:0/1356472141 conn(0x7fc7200fa100 msgr2=0x7fc7200fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:27.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 -- 172.21.15.64:0/1356472141 shutdown_connections 2023-07-14T18:24:27.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.817+0000 7fc7264e5700 1 -- 172.21.15.64:0/1356472141 wait complete. 2023-07-14T18:24:27.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 Processor -- start 2023-07-14T18:24:27.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 -- start start 2023-07-14T18:24:27.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7200feb00 0x7fc7201947b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 0x7fc720194cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72019c330 con 0x7fc720100680 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72019c470 con 0x7fc7200ffcf0 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.818+0000 7fc7264e5700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72019c600 con 0x7fc7200feb00 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48062/0 (socket says 172.21.15.64:48062) 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 -- 172.21.15.64:0/1224961247 learned_addr learned my addr 172.21.15.64:0/1224961247 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc71f7fe700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 0x7fc720194cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc71ffff700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7200feb00 0x7fc7201947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 -- 172.21.15.64:0/1224961247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7200feb00 msgr2=0x7fc7201947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7200feb00 0x7fc7201947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 -- 172.21.15.64:0/1224961247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 msgr2=0x7fc720194cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 0x7fc720194cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 -- 172.21.15.64:0/1224961247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc714009520 con 0x7fc720100680 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc724a82700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc71800ef30 tx=0x7fc71800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.875 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718004bb0 con 0x7fc720100680 2023-07-14T18:24:27.876 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7100095f0 con 0x7fc720100680 2023-07-14T18:24:27.876 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc71800f040 con 0x7fc720100680 2023-07-14T18:24:27.876 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc718004020 con 0x7fc720100680 2023-07-14T18:24:27.876 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.819+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc720067b80 con 0x7fc720100680 2023-07-14T18:24:27.876 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.820+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc720068570 con 0x7fc720100680 2023-07-14T18:24:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.821+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc718003a00 con 0x7fc720100680 2023-07-14T18:24:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.822+0000 7fc71d7fa700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 0x7fc7080a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.822+0000 7fc71ffff700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 0x7fc7080a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.822+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc718016070 con 0x7fc720100680 2023-07-14T18:24:27.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.822+0000 7fc71ffff700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 0x7fc7080a36d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7100098e0 tx=0x7fc71000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:27.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:27.825+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc718097ee0 con 0x7fc720100680 2023-07-14T18:24:28.078 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.026+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc720105ac0 con 0x7fc720100680 2023-07-14T18:24:28.080 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.028+0000 7fc71d7fa700 1 -- 172.21.15.64:0/1224961247 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7fc718003cb0 con 0x7fc720100680 2023-07-14T18:24:28.081 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:28.081 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:28.082 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 msgr2=0x7fc7080a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 0x7fc7080a36d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc7100098e0 tx=0x7fc71000b260 comp rx=0 tx=0).stop 2023-07-14T18:24:28.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 msgr2=0x7fc72019bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc71800ef30 tx=0x7fc71800c3a0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.083 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 shutdown_connections 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc7200feb00 0x7fc7201947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc7080a1220 0x7fc7080a36d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc720100680 0x7fc72019bd60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 --2- 172.21.15.64:0/1224961247 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7200ffcf0 0x7fc720194cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 >> 172.21.15.64:0/1224961247 conn(0x7fc7200fa100 msgr2=0x7fc720103220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.084 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 shutdown_connections 2023-07-14T18:24:28.085 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.030+0000 7fc7264e5700 1 -- 172.21.15.64:0/1224961247 wait complete. 2023-07-14T18:24:28.095 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:24:28.096 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:24:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 -- 172.21.15.64:0/2467457061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c102fb0 msgr2=0x7fbf5c105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 --2- 172.21.15.64:0/2467457061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c105390 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c009990 tx=0x7fbf4c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 -- 172.21.15.64:0/2467457061 shutdown_connections 2023-07-14T18:24:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 --2- 172.21.15.64:0/2467457061 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c100690 0x7fbf5c102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.226 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 --2- 172.21.15.64:0/2467457061 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 --2- 172.21.15.64:0/2467457061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c105390 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 -- 172.21.15.64:0/2467457061 >> 172.21.15.64:0/2467457061 conn(0x7fbf5c0fa130 msgr2=0x7fbf5c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 -- 172.21.15.64:0/2467457061 shutdown_connections 2023-07-14T18:24:28.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.173+0000 7fbf61521700 1 -- 172.21.15.64:0/2467457061 wait complete. 2023-07-14T18:24:28.227 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 Processor -- start 2023-07-14T18:24:28.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 -- start start 2023-07-14T18:24:28.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c100690 0x7fbf5c100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.229 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c0fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0ff320 con 0x7fbf5c1058d0 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0ff490 con 0x7fbf5c100690 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf61521700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c19f8c0 con 0x7fbf5c102fb0 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48076/0 (socket says 172.21.15.64:48076) 2023-07-14T18:24:28.230 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 -- 172.21.15.64:0/2770420338 learned_addr learned my addr 172.21.15.64:0/2770420338 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5affd700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c100690 0x7fbf5c100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5a7fc700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c0fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 -- 172.21.15.64:0/2770420338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c102fb0 msgr2=0x7fbf5c0fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c0fe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 -- 172.21.15.64:0/2770420338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c100690 msgr2=0x7fbf5c100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c100690 0x7fbf5c100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.174+0000 7fbf5b7fe700 1 -- 172.21.15.64:0/2770420338 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c0095f0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.231 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf5b7fe700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007ee0 tx=0x7fbf5400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54017070 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbf5400f040 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c19fba0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54005710 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.175+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c1a0060 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.176+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c0fbca0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.232 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.177+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbf54005870 con 0x7fbf5c1058d0 2023-07-14T18:24:28.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.178+0000 7fbf43fff700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 0x7fbf440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.233 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.178+0000 7fbf5affd700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 0x7fbf440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.178+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fbf540ccbc0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.178+0000 7fbf5affd700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 0x7fbf440a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf50005950 tx=0x7fbf5000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.234 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.181+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbf54095ea0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.382+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf5c02cd70 con 0x7fbf5c1058d0 2023-07-14T18:24:28.435 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.383+0000 7fbf43fff700 1 -- 172.21.15.64:0/2770420338 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf540994c0 con 0x7fbf5c1058d0 2023-07-14T18:24:28.436 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:24:28.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 msgr2=0x7fbf440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 0x7fbf440a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbf50005950 tx=0x7fbf5000b410 comp rx=0 tx=0).stop 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 msgr2=0x7fbf5c0feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007ee0 tx=0x7fbf5400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 shutdown_connections 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fbf440a11e0 0x7fbf440a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf5c1058d0 0x7fbf5c0feb80 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbf5c100690 0x7fbf5c100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 --2- 172.21.15.64:0/2770420338 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fbf5c102fb0 0x7fbf5c0fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 >> 172.21.15.64:0/2770420338 conn(0x7fbf5c0fa130 msgr2=0x7fbf5c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 shutdown_connections 2023-07-14T18:24:28.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.385+0000 7fbf61521700 1 -- 172.21.15.64:0/2770420338 wait complete. 2023-07-14T18:24:28.449 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:28 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1735388068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:28.449 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:28 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3891732827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.450 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:28 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1224961247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:24:28.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 -- 172.21.15.64:0/1286398667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0ffd00 msgr2=0x7f124c100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 --2- 172.21.15.64:0/1286398667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0ffd00 0x7f124c100150 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1234009990 tx=0x7f1234009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 -- 172.21.15.64:0/1286398667 shutdown_connections 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 --2- 172.21.15.64:0/1286398667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c0feb10 0x7f124c0fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 --2- 172.21.15.64:0/1286398667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c100690 0x7f124c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 --2- 172.21.15.64:0/1286398667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0ffd00 0x7f124c100150 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 -- 172.21.15.64:0/1286398667 >> 172.21.15.64:0/1286398667 conn(0x7f124c0fa130 msgr2=0x7f124c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 -- 172.21.15.64:0/1286398667 shutdown_connections 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.527+0000 7f12514c4700 1 -- 172.21.15.64:0/1286398667 wait complete. 2023-07-14T18:24:28.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 Processor -- start 2023-07-14T18:24:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 -- start start 2023-07-14T18:24:28.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 0x7f124c1010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 0x7f124c101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c101dc0 con 0x7f124c0feb10 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c101f30 con 0x7f124c0ffd00 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f12514c4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c19f8e0 con 0x7f124c100690 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f124affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f124affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48086/0 (socket says 172.21.15.64:48086) 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.528+0000 7f124affd700 1 -- 172.21.15.64:0/4142961777 learned_addr learned my addr 172.21.15.64:0/4142961777 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f12427fc700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 0x7f124c1010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124b7fe700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 0x7f124c101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 -- 172.21.15.64:0/4142961777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 msgr2=0x7f124c101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 0x7f124c101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 -- 172.21.15.64:0/4142961777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 msgr2=0x7f124c1010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.585 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 0x7f124c1010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 -- 172.21.15.64:0/4142961777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12340095f0 con 0x7f124c0feb10 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f12427fc700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 0x7f124c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124b7fe700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 0x7f124c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f124affd700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e970 tx=0x7f123c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c009790 con 0x7f124c0feb10 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f123c0044b0 con 0x7f124c0feb10 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c003dd0 con 0x7f124c0feb10 2023-07-14T18:24:28.586 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c19fbc0 con 0x7f124c0feb10 2023-07-14T18:24:28.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.529+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c1a00d0 con 0x7f124c0feb10 2023-07-14T18:24:28.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.530+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c10a190 con 0x7f124c0feb10 2023-07-14T18:24:28.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.531+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f123c003f30 con 0x7f124c0feb10 2023-07-14T18:24:28.587 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.532+0000 7f1248ff9700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 0x7f12380a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.532+0000 7f12427fc700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 0x7f12380a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.532+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f123c014070 con 0x7f124c0feb10 2023-07-14T18:24:28.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.532+0000 7f12427fc700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 0x7f12380a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f123400b2e0 tx=0x7f1234005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.588 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.536+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f123c096360 con 0x7f124c0feb10 2023-07-14T18:24:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:28 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1735388068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:28 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3891732827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:28 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1224961247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.737+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f124c1a06e0 con 0x7f124c0feb10 2023-07-14T18:24:28.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.737+0000 7f1248ff9700 1 -- 172.21.15.64:0/4142961777 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f123c099980 con 0x7f124c0feb10 2023-07-14T18:24:28.790 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:28.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 msgr2=0x7f12380a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 0x7f12380a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f123400b2e0 tx=0x7f1234005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 msgr2=0x7f124c102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f123c00e970 tx=0x7f123c00ed30 comp rx=0 tx=0).stop 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 shutdown_connections 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f124c100690 0x7f124c101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f12380a1260 0x7f12380a3710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f124c0feb10 0x7f124c102aa0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 --2- 172.21.15.64:0/4142961777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f124c0ffd00 0x7f124c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.739+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 >> 172.21.15.64:0/4142961777 conn(0x7f124c0fa130 msgr2=0x7f124c0677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.740+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 shutdown_connections 2023-07-14T18:24:28.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.740+0000 7f12514c4700 1 -- 172.21.15.64:0/4142961777 wait complete. 2023-07-14T18:24:28.805 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:24:28.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:28 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1735388068' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:28.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:28 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3891732827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:28 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1224961247' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- 172.21.15.64:0/4086599843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c101790 msgr2=0x7f998c108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 --2- 172.21.15.64:0/4086599843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c101790 0x7f998c108050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f998400b0a0 tx=0x7f998400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- 172.21.15.64:0/4086599843 shutdown_connections 2023-07-14T18:24:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 --2- 172.21.15.64:0/4086599843 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c0ffc10 0x7f998c100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.937 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 --2- 172.21.15.64:0/4086599843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c101790 0x7f998c108050 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 --2- 172.21.15.64:0/4086599843 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c100e00 0x7f998c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- 172.21.15.64:0/4086599843 >> 172.21.15.64:0/4086599843 conn(0x7f998c0fb250 msgr2=0x7f998c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- 172.21.15.64:0/4086599843 shutdown_connections 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- 172.21.15.64:0/4086599843 wait complete. 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 Processor -- start 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.882+0000 7f9992994700 1 -- start start 2023-07-14T18:24:28.938 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c100e00 0x7f998c198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 0x7f998c19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c19f740 con 0x7f998c0ffc10 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c19f880 con 0x7f998c101790 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f998c19fa10 con 0x7f998c100e00 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48100/0 (socket says 172.21.15.64:48100) 2023-07-14T18:24:28.939 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 -- 172.21.15.64:0/3689607486 learned_addr learned my addr 172.21.15.64:0/3689607486 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9990f31700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 0x7f998c19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998b7fe700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c100e00 0x7f998c198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 -- 172.21.15.64:0/3689607486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c100e00 msgr2=0x7f998c198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c100e00 0x7f998c198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 -- 172.21.15.64:0/3689607486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 msgr2=0x7f998c19f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 0x7f998c19f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:28.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 -- 172.21.15.64:0/3689607486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9984009d00 con 0x7f998c0ffc10 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9990f31700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 0x7f998c19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f998bfff700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c970 tx=0x7f997c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c004220 con 0x7f998c0ffc10 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f997c007d30 con 0x7f998c0ffc10 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997c003bf0 con 0x7f998c0ffc10 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.883+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c19fcf0 con 0x7f998c0ffc10 2023-07-14T18:24:28.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.884+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c1a01b0 con 0x7f998c0ffc10 2023-07-14T18:24:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.884+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c10b2f0 con 0x7f998c0ffc10 2023-07-14T18:24:28.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.886+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f997c003d70 con 0x7f998c0ffc10 2023-07-14T18:24:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.886+0000 7f99897fa700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 0x7f99740a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.886+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f997c0cce50 con 0x7f998c0ffc10 2023-07-14T18:24:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.886+0000 7f998b7fe700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 0x7f99740a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.887+0000 7f998b7fe700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 0x7f99740a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9980005fd0 tx=0x7f998000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:28.943 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:28.890+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f997c096130 con 0x7f998c0ffc10 2023-07-14T18:24:29.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.089+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f998c1a0760 con 0x7f998c0ffc10 2023-07-14T18:24:29.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.089+0000 7f99897fa700 1 -- 172.21.15.64:0/3689607486 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f997c01d0a0 con 0x7f998c0ffc10 2023-07-14T18:24:29.143 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:24:29.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 msgr2=0x7f99740a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 0x7f99740a3690 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9980005fd0 tx=0x7f998000a560 comp rx=0 tx=0).stop 2023-07-14T18:24:29.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 msgr2=0x7f998c197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f997c00c970 tx=0x7f997c00cd30 comp rx=0 tx=0).stop 2023-07-14T18:24:29.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 shutdown_connections 2023-07-14T18:24:29.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f99740a11e0 0x7f99740a3690 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f998c101790 0x7f998c19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f998c0ffc10 0x7f998c197bc0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 --2- 172.21.15.64:0/3689607486 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f998c100e00 0x7f998c198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.091+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 >> 172.21.15.64:0/3689607486 conn(0x7f998c0fb250 msgr2=0x7f998c0fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:29.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.092+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 shutdown_connections 2023-07-14T18:24:29.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.092+0000 7f9992994700 1 -- 172.21.15.64:0/3689607486 wait complete. 2023-07-14T18:24:29.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:24:29.290 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 -- 172.21.15.64:0/3760861200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878074620 msgr2=0x7f7878072a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 --2- 172.21.15.64:0/3760861200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878074620 0x7f7878072a70 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7868009990 tx=0x7f7868009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 -- 172.21.15.64:0/3760861200 shutdown_connections 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 --2- 172.21.15.64:0/3760861200 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878107ab0 0x7f7878109f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 --2- 172.21.15.64:0/3760861200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878074620 0x7f7878072a70 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 --2- 172.21.15.64:0/3760861200 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 0x7f78780734b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 -- 172.21.15.64:0/3760861200 >> 172.21.15.64:0/3760861200 conn(0x7f78780fb260 msgr2=0x7f78780fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 -- 172.21.15.64:0/3760861200 shutdown_connections 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.235+0000 7f787f91b700 1 -- 172.21.15.64:0/3760861200 wait complete. 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 Processor -- start 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 -- start start 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 0x7f7878197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878074620 0x7f7878197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787819f5f0 con 0x7f7878107ab0 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787819f730 con 0x7f7878074620 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787f91b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787819f8c0 con 0x7f7878073040 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48116/0 (socket says 172.21.15.64:48116) 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 -- 172.21.15.64:0/3055266797 learned_addr learned my addr 172.21.15.64:0/3055266797 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:29.293 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787d6b7700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 0x7f7878197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787ceb6700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878074620 0x7f7878197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 -- 172.21.15.64:0/3055266797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 msgr2=0x7f7878197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 0x7f7878197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 -- 172.21.15.64:0/3055266797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878074620 msgr2=0x7f7878197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878074620 0x7f7878197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.236+0000 7f787deb8700 1 -- 172.21.15.64:0/3055266797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78680095f0 con 0x7f7878107ab0 2023-07-14T18:24:29.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f787deb8700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7870007ee0 tx=0x7f787000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870017070 con 0x7f7878107ab0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f787000f040 con 0x7f7878107ab0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787819fba0 con 0x7f7878107ab0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7870005570 con 0x7f7878107ab0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.237+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78781a00b0 con 0x7f7878107ab0 2023-07-14T18:24:29.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.238+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78781061f0 con 0x7f7878107ab0 2023-07-14T18:24:29.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.239+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f787000db20 con 0x7f7878107ab0 2023-07-14T18:24:29.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.239+0000 7f786e7fc700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 0x7f78640a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.240+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f78700cc510 con 0x7f7878107ab0 2023-07-14T18:24:29.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.240+0000 7f787d6b7700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 0x7f78640a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.240+0000 7f787d6b7700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 0x7f78640a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f786800b5c0 tx=0x7f7868005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:29.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.244+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7870095790 con 0x7f7878107ab0 2023-07-14T18:24:29.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.445+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f787802cb00 con 0x7f7878107ab0 2023-07-14T18:24:29.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.446+0000 7f786e7fc700 1 -- 172.21.15.64:0/3055266797 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7870098db0 con 0x7f7878107ab0 2023-07-14T18:24:29.499 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:29.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 msgr2=0x7f78640a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 0x7f78640a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f786800b5c0 tx=0x7f7868005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 msgr2=0x7f787819f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7870007ee0 tx=0x7f787000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 shutdown_connections 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7878073040 0x7f7878197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f78640a1260 0x7f78640a3710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7878107ab0 0x7f787819f020 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 --2- 172.21.15.64:0/3055266797 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7878074620 0x7f7878197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 >> 172.21.15.64:0/3055266797 conn(0x7f78780fb260 msgr2=0x7f78780fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 shutdown_connections 2023-07-14T18:24:29.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.448+0000 7f787f91b700 1 -- 172.21.15.64:0/3055266797 wait complete. 2023-07-14T18:24:29.512 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:29 smithi064 ceph-mon[96720]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2023-07-14T18:24:29.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2770420338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4142961777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.513 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:29 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3689607486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:24:29.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.587+0000 7fae1816c700 1 -- 172.21.15.64:0/644471883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae100ffcf0 msgr2=0x7fae10100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.587+0000 7fae1816c700 1 --2- 172.21.15.64:0/644471883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae100ffcf0 0x7fae10100140 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fae00009990 tx=0x7fae00009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.587+0000 7fae1816c700 1 -- 172.21.15.64:0/644471883 shutdown_connections 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.587+0000 7fae1816c700 1 --2- 172.21.15.64:0/644471883 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae10100680 0x7fae10106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 --2- 172.21.15.64:0/644471883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100feb00 0x7fae100fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 --2- 172.21.15.64:0/644471883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae100ffcf0 0x7fae10100140 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.641 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- 172.21.15.64:0/644471883 >> 172.21.15.64:0/644471883 conn(0x7fae100fa100 msgr2=0x7fae100fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:29.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- 172.21.15.64:0/644471883 shutdown_connections 2023-07-14T18:24:29.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- 172.21.15.64:0/644471883 wait complete. 2023-07-14T18:24:29.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 Processor -- start 2023-07-14T18:24:29.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- start start 2023-07-14T18:24:29.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae100feb00 0x7fae10194730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100ffcf0 0x7fae10194c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019c2b0 con 0x7fae10100680 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019c3f0 con 0x7fae100ffcf0 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.588+0000 7fae1816c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019c580 con 0x7fae100feb00 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.644 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48130/0 (socket says 172.21.15.64:48130) 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 -- 172.21.15.64:0/2098060547 learned_addr learned my addr 172.21.15.64:0/2098060547 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae15707700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100ffcf0 0x7fae10194c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae15f08700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae100feb00 0x7fae10194730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 -- 172.21.15.64:0/2098060547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae100feb00 msgr2=0x7fae10194730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae100feb00 0x7fae10194730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 -- 172.21.15.64:0/2098060547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100ffcf0 msgr2=0x7fae10194c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100ffcf0 0x7fae10194c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 -- 172.21.15.64:0/2098060547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000095f0 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae16709700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fae08007d70 tx=0x7fae0800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800ad40 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae100678c0 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fae080128b0 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae080116b0 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.589+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae10067d80 con 0x7fae10100680 2023-07-14T18:24:29.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.590+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1010a180 con 0x7fae10100680 2023-07-14T18:24:29.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.592+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fae08012a20 con 0x7fae10100680 2023-07-14T18:24:29.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.592+0000 7fae06ffd700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 0x7fadfc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:29.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.592+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fae080cd330 con 0x7fae10100680 2023-07-14T18:24:29.647 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.592+0000 7fae15f08700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 0x7fadfc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:29.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.593+0000 7fae15f08700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 0x7fadfc0a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fae100ffb50 tx=0x7fae0c00bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:29.648 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.596+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fae08096720 con 0x7fae10100680 2023-07-14T18:24:29.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:29 smithi144 ceph-mon[98417]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2023-07-14T18:24:29.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:29 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2770420338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:29 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4142961777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:29 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3689607486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:29 smithi105 ceph-mon[98013]: pgmap v298: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9 op/s 2023-07-14T18:24:29.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:29 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2770420338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:29 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4142961777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:29 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3689607486' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:29.891 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.838+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae10105e40 con 0x7fae10100680 2023-07-14T18:24:29.892 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.839+0000 7fae06ffd700 1 -- 172.21.15.64:0/2098060547 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7fae08099d40 con 0x7fae10100680 2023-07-14T18:24:29.893 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:29.893 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:24:29.895 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 msgr2=0x7fadfc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 0x7fadfc0a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fae100ffb50 tx=0x7fae0c00bef0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 msgr2=0x7fae1019bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:29.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fae08007d70 tx=0x7fae0800a340 comp rx=0 tx=0).stop 2023-07-14T18:24:29.896 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 shutdown_connections 2023-07-14T18:24:29.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fae100feb00 0x7fae10194730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fadfc0a1220 0x7fadfc0a36d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.897 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae10100680 0x7fae1019bce0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 --2- 172.21.15.64:0/2098060547 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fae100ffcf0 0x7fae10194c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:29.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 >> 172.21.15.64:0/2098060547 conn(0x7fae100fa100 msgr2=0x7fae10103180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:29.898 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 shutdown_connections 2023-07-14T18:24:29.899 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.841+0000 7fae1816c700 1 -- 172.21.15.64:0/2098060547 wait complete. 2023-07-14T18:24:29.899 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:29.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:30.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 -- 172.21.15.64:0/452462487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 msgr2=0x7f8920100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 --2- 172.21.15.64:0/452462487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920100140 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f8908009990 tx=0x7f8908009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 -- 172.21.15.64:0/452462487 shutdown_connections 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 --2- 172.21.15.64:0/452462487 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 0x7f89200fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 --2- 172.21.15.64:0/452462487 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 0x7f8920106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 --2- 172.21.15.64:0/452462487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920100140 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 -- 172.21.15.64:0/452462487 >> 172.21.15.64:0/452462487 conn(0x7f89200fa100 msgr2=0x7f89200fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.980+0000 7f8925209700 1 -- 172.21.15.64:0/452462487 shutdown_connections 2023-07-14T18:24:30.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 -- 172.21.15.64:0/452462487 wait complete. 2023-07-14T18:24:30.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 Processor -- start 2023-07-14T18:24:30.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 -- start start 2023-07-14T18:24:30.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 0x7f8920194710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 0x7f892019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892019c290 con 0x7f89200ffcf0 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892019c3d0 con 0x7f8920100680 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.981+0000 7f8925209700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892019c560 con 0x7f89200feb00 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48140/0 (socket says 172.21.15.64:48140) 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 -- 172.21.15.64:0/3682392887 learned_addr learned my addr 172.21.15.64:0/3682392887 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f891ed9d700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 0x7f8920194710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f891f59e700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 0x7f892019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 -- 172.21.15.64:0/3682392887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 msgr2=0x7f8920194710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 0x7f8920194710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 -- 172.21.15.64:0/3682392887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 msgr2=0x7f892019bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 0x7f892019bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 -- 172.21.15.64:0/3682392887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89080095f0 con 0x7f89200ffcf0 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8917fff700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8908003ec0 tx=0x7f8908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f890801d070 con 0x7f89200ffcf0 2023-07-14T18:24:30.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f890800bd50 con 0x7f89200ffcf0 2023-07-14T18:24:30.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892019c7e0 con 0x7f89200ffcf0 2023-07-14T18:24:30.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8908022d40 con 0x7f89200ffcf0 2023-07-14T18:24:30.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.982+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892019cc00 con 0x7f89200ffcf0 2023-07-14T18:24:30.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.983+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892010a180 con 0x7f89200ffcf0 2023-07-14T18:24:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.984+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8908022470 con 0x7f89200ffcf0 2023-07-14T18:24:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.985+0000 7f891cd99700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 0x7f890c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.985+0000 7f891ed9d700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 0x7f890c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.985+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f89080ce2e0 con 0x7f89200ffcf0 2023-07-14T18:24:30.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.986+0000 7f891ed9d700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 0x7f890c0a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89200ffb50 tx=0x7f8910009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:29.989+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8908097650 con 0x7f89200ffcf0 2023-07-14T18:24:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.188+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8920105a50 con 0x7f89200ffcf0 2023-07-14T18:24:30.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.190+0000 7f891cd99700 1 -- 172.21.15.64:0/3682392887 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7f890809ac70 con 0x7f89200ffcf0 2023-07-14T18:24:30.243 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:30.243 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 msgr2=0x7f890c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 0x7f890c0a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89200ffb50 tx=0x7f8910009260 comp rx=0 tx=0).stop 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 msgr2=0x7f8920194c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8908003ec0 tx=0x7f8908004730 comp rx=0 tx=0).stop 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 shutdown_connections 2023-07-14T18:24:30.245 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f89200feb00 0x7f8920194710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f890c0a1220 0x7f890c0a36d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8920100680 0x7f892019bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 --2- 172.21.15.64:0/3682392887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f89200ffcf0 0x7f8920194c50 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 >> 172.21.15.64:0/3682392887 conn(0x7f89200fa100 msgr2=0x7f89200fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 shutdown_connections 2023-07-14T18:24:30.246 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.192+0000 7f8925209700 1 -- 172.21.15.64:0/3682392887 wait complete. 2023-07-14T18:24:30.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:24:30.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 -- 172.21.15.64:0/4170091625 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0103040 msgr2=0x7fafb0105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 --2- 172.21.15.64:0/4170091625 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0103040 0x7fafb0105420 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7faf98009990 tx=0x7faf98009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 -- 172.21.15.64:0/4170091625 shutdown_connections 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 --2- 172.21.15.64:0/4170091625 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0105960 0x7fafb0107d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 --2- 172.21.15.64:0/4170091625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0100720 0x7fafb0102b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 --2- 172.21.15.64:0/4170091625 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0103040 0x7fafb0105420 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 -- 172.21.15.64:0/4170091625 >> 172.21.15.64:0/4170091625 conn(0x7fafb00fa100 msgr2=0x7fafb00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 -- 172.21.15.64:0/4170091625 shutdown_connections 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.332+0000 7fafb5574700 1 -- 172.21.15.64:0/4170091625 wait complete. 2023-07-14T18:24:30.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 Processor -- start 2023-07-14T18:24:30.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 -- start start 2023-07-14T18:24:30.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0100720 0x7fafb0100090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.388 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0103040 0x7fafb00fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00ff3b0 con 0x7fafb0105960 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb00ff520 con 0x7fafb0103040 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafb5574700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb019f830 con 0x7fafb0100720 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48156/0 (socket says 172.21.15.64:48156) 2023-07-14T18:24:30.389 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafaf7fe700 1 -- 172.21.15.64:0/1243986030 learned_addr learned my addr 172.21.15.64:0/1243986030 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.333+0000 7fafae7fc700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0103040 0x7fafb00fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaeffd700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0100720 0x7fafb0100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 -- 172.21.15.64:0/1243986030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0100720 msgr2=0x7fafb0100090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0100720 0x7fafb0100090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 -- 172.21.15.64:0/1243986030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0103040 msgr2=0x7fafb00fe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0103040 0x7fafb00fe6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.390 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 -- 172.21.15.64:0/1243986030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980095f0 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafaf7fe700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fafa8007ee0 tx=0x7fafa800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8017070 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fafa800f040 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb019fb10 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8005710 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.334+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb019ffd0 con 0x7fafb0105960 2023-07-14T18:24:30.391 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.335+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb010b260 con 0x7fafb0105960 2023-07-14T18:24:30.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.336+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fafa8005870 con 0x7fafb0105960 2023-07-14T18:24:30.392 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.337+0000 7fafa7fff700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 0x7faf9c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.337+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fafa80cc620 con 0x7fafb0105960 2023-07-14T18:24:30.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.337+0000 7fafaeffd700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 0x7faf9c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.337+0000 7fafaeffd700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 0x7faf9c0a36d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fafa0005950 tx=0x7fafa00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.393 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.341+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fafa80958a0 con 0x7fafb0105960 2023-07-14T18:24:30.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:30 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3055266797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:30.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:30 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2098060547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:30 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3682392887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.587+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafb010b450 con 0x7fafb0105960 2023-07-14T18:24:30.640 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.587+0000 7fafa7fff700 1 -- 172.21.15.64:0/1243986030 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7fafa8095020 con 0x7fafb0105960 2023-07-14T18:24:30.640 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:30.640 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:24:30.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 msgr2=0x7faf9c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 0x7faf9c0a36d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fafa0005950 tx=0x7fafa00058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 msgr2=0x7fafb00fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fafa8007ee0 tx=0x7fafa800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.642 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 shutdown_connections 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fafb0100720 0x7fafb0100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faf9c0a1220 0x7faf9c0a36d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fafb0105960 0x7fafb00fec10 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 --2- 172.21.15.64:0/1243986030 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fafb0103040 0x7fafb00fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 >> 172.21.15.64:0/1243986030 conn(0x7fafb00fa100 msgr2=0x7fafb0101ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 shutdown_connections 2023-07-14T18:24:30.643 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.589+0000 7fafb5574700 1 -- 172.21.15.64:0/1243986030 wait complete. 2023-07-14T18:24:30.644 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:30.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T18:23:04.865157+0000', 'modified': '2023-07-14T18:23:28.278497+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24545, 'mds_1': 24508, 'mds_2': 24500, 'mds_3': 24487, 'mds_4': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14550': {'gid': 14550, 'name': 'j', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/2756017867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 2756017867}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 2756017867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14556': {'gid': 14556, 'name': 'd', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6837/1035088143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 1035088143}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 1035088143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6839/3820392799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 3820392799}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 3820392799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6841/146209694', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6840', 'nonce': 146209694}, {'type': 'v1', 'addr': '172.21.15.64:6841', 'nonce': 146209694}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6835/2365162845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 2365162845}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 2365162845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.144:6835/4223527896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6834', 'nonce': 4223527896}, {'type': 'v1', 'addr': '172.21.15.144:6835', 'nonce': 4223527896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.144:6837/1582348642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6836', 'nonce': 1582348642}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 1582348642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24500': {'gid': 24500, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.105:6837/2898950423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 2898950423}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 2898950423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.144:6839/4051776603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6838', 'nonce': 4051776603}, {'type': 'v1', 'addr': '172.21.15.144:6839', 'nonce': 4051776603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/2402376208', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 2402376208}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 2402376208}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T18:24:30.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T18:24:30.655 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-07-14T18:24:30.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:30 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3055266797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:30.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:30 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2098060547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:30 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3682392887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.782 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2154773452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc1006c0 msgr2=0x7ffabc102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2154773452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc1006c0 0x7ffabc102aa0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009990 tx=0x7ffaac009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2154773452 shutdown_connections 2023-07-14T18:24:30.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2154773452 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc105900 0x7ffabc107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2154773452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc1006c0 0x7ffabc102aa0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.783 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2154773452 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc102fe0 0x7ffabc1053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2154773452 >> 172.21.15.64:0/2154773452 conn(0x7ffabc0fa160 msgr2=0x7ffabc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2154773452 shutdown_connections 2023-07-14T18:24:30.784 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.730+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2154773452 wait complete. 2023-07-14T18:24:30.785 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 Processor -- start 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 -- start start 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc1006c0 0x7ffabc195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 0x7ffabc195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc19d3f0 con 0x7ffabc105900 2023-07-14T18:24:30.786 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc19d530 con 0x7ffabc1006c0 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffac0c0e700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffabc19d6c0 con 0x7ffabc102fe0 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48170/0 (socket says 172.21.15.64:48170) 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 -- 172.21.15.64:0/2408798333 learned_addr learned my addr 172.21.15.64:0/2408798333 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc1006c0 0x7ffabc195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffab3fff700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 0x7ffabc195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 -- 172.21.15.64:0/2408798333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 msgr2=0x7ffabc195db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.787 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 0x7ffabc195db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 -- 172.21.15.64:0/2408798333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc1006c0 msgr2=0x7ffabc195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc1006c0 0x7ffabc195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 -- 172.21.15.64:0/2408798333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac0095f0 con 0x7ffabc105900 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffab3fff700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 0x7ffabc195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.731+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eb80 tx=0x7ffab400ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.788 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.732+0000 7ffab37fe700 1 -- 172.21.15.64:0/2408798333 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4018070 con 0x7ffabc105900 2023-07-14T18:24:30.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.732+0000 7ffab37fe700 1 -- 172.21.15.64:0/2408798333 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffab400f040 con 0x7ffabc105900 2023-07-14T18:24:30.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.732+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffabc19d940 con 0x7ffabc105900 2023-07-14T18:24:30.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.732+0000 7ffab37fe700 1 -- 172.21.15.64:0/2408798333 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400cdf0 con 0x7ffabc105900 2023-07-14T18:24:30.789 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.732+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc19ddf0 con 0x7ffabc105900 2023-07-14T18:24:30.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.734+0000 7ffab37fe700 1 -- 172.21.15.64:0/2408798333 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ffab4003a50 con 0x7ffabc105900 2023-07-14T18:24:30.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.734+0000 7ffab37fe700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 0x7ffaa80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.734+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 0x7ffaa80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.734+0000 7ffab37fe700 1 -- 172.21.15.64:0/2408798333 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffab40cc940 con 0x7ffabc105900 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.735+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 0x7ffaa80a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffaac006010 tx=0x7ffaac00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.735+0000 7ffaccda6b80 1 Processor -- start 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.735+0000 7ffaccda6b80 1 -- start start 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.735+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb0a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.735+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb0ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fa0ec60 con 0x56061fb0b460 2023-07-14T18:24:30.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fa619f0 con 0x56061fb09f10 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fa5ae80 con 0x56061fb0a860 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48178/0 (socket says 172.21.15.64:48178) 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 -- 172.21.15.64:0/8135784 learned_addr learned my addr 172.21.15.64:0/8135784 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffab3fff700 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb0ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffaba59c700 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb0a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 -- 172.21.15.64:0/8135784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 msgr2=0x56061fb0ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb0ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 -- 172.21.15.64:0/8135784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 msgr2=0x56061fb0a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb0a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 -- 172.21.15.64:0/8135784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56061f888d90 con 0x56061fb0b460 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffabad9d700 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffab4099670 tx=0x7ffab4099580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ec2278b3e7f959 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffab0ff9700 1 -- 172.21.15.64:0/8135784 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab409d9a0 con 0x56061fb0b460 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffab0ff9700 1 -- 172.21.15.64:0/8135784 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffab40a1d30 con 0x56061fb0b460 2023-07-14T18:24:30.797 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.736+0000 7ffab0ff9700 1 -- 172.21.15.64:0/8135784 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4098e60 con 0x56061fb0b460 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- 172.21.15.64:0/8135784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 msgr2=0x56061fb0b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffab4099670 tx=0x7ffab4099580 comp rx=0 tx=0).stop 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- 172.21.15.64:0/8135784 shutdown_connections 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb0b8d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb0a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/8135784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb0ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.798 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- 172.21.15.64:0/8135784 >> 172.21.15.64:0/8135784 conn(0x56061faff630 msgr2=0x56061fb04980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- 172.21.15.64:0/8135784 shutdown_connections 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- 172.21.15.64:0/8135784 wait complete. 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.737+0000 7ffaccda6b80 1 -- start start 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 Processor -- start 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb13e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb1aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061f9faa70 con 0x56061fb09f10 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061f9fa770 con 0x56061fb0b460 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061f8de8d0 con 0x56061fb0a860 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaccda6b80 10 client.0 _ll_register_callbacks cb 0x56061f9f8710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48180/0 (socket says 172.21.15.64:48180) 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 -- 172.21.15.64:0/861982643 learned_addr learned my addr 172.21.15.64:0/861982643 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffab3fff700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb13e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 -- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 msgr2=0x56061fb13e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb13e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 -- 172.21.15.64:0/861982643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 msgr2=0x56061fb1aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffabad9d700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb1aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb1aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 -- 172.21.15.64:0/861982643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffabc19ddf0 con 0x56061fb09f10 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffabad9d700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb1aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:30.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaba59c700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009990 tx=0x7ffaac009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6b9c9a064177c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.738+0000 7ffaa1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac00bba0 con 0x56061fb09f10 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffaac00bd00 con 0x56061fb09f10 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56061f9b44a0 con 0x56061fb09f10 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffaac00bec0 con 0x56061fb09f10 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 10 client.15231 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56061f871c10 con 0x56061fb09f10 2023-07-14T18:24:30.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x56061f9b44a0 con 0x56061fb09f10 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffaac00f580 con 0x56061fb09f10 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 10 client.15231 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 10 client.15231 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 20 client.15231 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb1ebd0 0x56061fb1dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 4 client.15231 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 --> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56061fad8d50 con 0x56061fb1ebd0 2023-07-14T18:24:30.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.739+0000 7ffab3fff700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb1ebd0 0x56061fb1dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.740+0000 7ffaa1ffb700 10 client.15231 ms_handle_connect on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.740+0000 7ffab3fff700 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb1ebd0 0x56061fb1dfd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=181d43fdf241c0d server_cookie=1295845a61f2e660 in_seq=0 out_seq=0 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.740+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/861982643 <== mds.4 v2:172.21.15.144:6834/4223527896 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x56061fad8d50 con 0x56061fb1ebd0 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.740+0000 7ffaa1ffb700 10 client.15231 handle_command_reply: tid=0 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 client.15231 shutdown 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 msgr2=0x56061fb13950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.804 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009990 tx=0x7ffaac009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 shutdown_connections 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb1ebd0 0x56061fb1dfd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb1aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaa1ffb700 0 client.15231 ms_handle_reset on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb09f10 0x56061fb13950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/861982643 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb13e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 >> 172.21.15.64:0/861982643 conn(0x56061faff630 msgr2=0x56061fb03600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 shutdown_connections 2023-07-14T18:24:30.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 1 -- 172.21.15.64:0/861982643 wait complete. 2023-07-14T18:24:30.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.741+0000 7ffaccda6b80 20 client.15231 trim_cache size 0 max 16384 2023-07-14T18:24:30.812 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.758+0000 7ffaccda6b80 1 Processor -- start 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.758+0000 7ffaccda6b80 1 -- start start 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.758+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 0x56061fb1dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.758+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb0e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fa29790 con 0x56061fb0a860 2023-07-14T18:24:30.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061f8eade0 con 0x56061fb0b460 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fa19480 con 0x56061fb09f10 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48190/0 (socket says 172.21.15.64:48190) 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 -- 172.21.15.64:0/4186752987 learned_addr learned my addr 172.21.15.64:0/4186752987 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaba59c700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 0x56061fb1dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffabad9d700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb0e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 -- 172.21.15.64:0/4186752987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 msgr2=0x56061fb1dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 0x56061fb1dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 -- 172.21.15.64:0/4186752987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 msgr2=0x56061fb0e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb0e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 -- 172.21.15.64:0/4186752987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56061f8af200 con 0x56061fb0a860 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffaba59c700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 0x56061fb1dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab3fff700 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x56061fb1fab0 tx=0x7ffaa40095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8cc477b33db6d8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab17fa700 1 -- 172.21.15.64:0/4186752987 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa4018070 con 0x56061fb0a860 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab17fa700 1 -- 172.21.15.64:0/4186752987 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffaa4019040 con 0x56061fb0a860 2023-07-14T18:24:30.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.759+0000 7ffab17fa700 1 -- 172.21.15.64:0/4186752987 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa401ac70 con 0x56061fb0a860 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- 172.21.15.64:0/4186752987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 msgr2=0x56061fb1e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x56061fb1fab0 tx=0x7ffaa40095a0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- 172.21.15.64:0/4186752987 shutdown_connections 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb09f10 0x56061fb1dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb0b460 0x56061fb0e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/4186752987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0a860 0x56061fb1e510 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- 172.21.15.64:0/4186752987 >> 172.21.15.64:0/4186752987 conn(0x56061faff630 msgr2=0x56061fb12620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- 172.21.15.64:0/4186752987 shutdown_connections 2023-07-14T18:24:30.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- 172.21.15.64:0/4186752987 wait complete. 2023-07-14T18:24:30.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 -- start start 2023-07-14T18:24:30.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.760+0000 7ffaccda6b80 1 Processor -- start 2023-07-14T18:24:30.818 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:30 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3055266797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:30.818 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:30 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2098060547' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.818 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:30 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3682392887' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:30.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb04aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb74860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fb77540 con 0x56061fb0b460 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fb776b0 con 0x56061fb09f10 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x56061fb77870 con 0x56061fb0a860 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 10 client.0 _ll_register_callbacks cb 0x7ffaac0057b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48198/0 (socket says 172.21.15.64:48198) 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 -- 172.21.15.64:0/2966736253 learned_addr learned my addr 172.21.15.64:0/2966736253 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb04aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffab3fff700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb74860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 -- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 msgr2=0x56061fb74860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb74860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 -- 172.21.15.64:0/2966736253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 msgr2=0x56061fb04aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb04aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 -- 172.21.15.64:0/2966736253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa4009bb0 con 0x56061fb0b460 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffabad9d700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a1800 tx=0x7ffab40098a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173022b755ae48fb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaa1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab409d9a0 con 0x56061fb0b460 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffab40ca230 con 0x56061fb0b460 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab40a1d30 con 0x56061fb0b460 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.761+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab4099bd0 con 0x56061fb0b460 2023-07-14T18:24:30.823 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 10 client.15237 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56061f887260 con 0x56061fb0b460 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ffab4063020 con 0x56061fb0b460 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffab401b9d0 con 0x56061fb0b460 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 10 client.15237 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 10 client.15237 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 20 client.15237 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:30.824 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb7c0b0 0x56061fb78b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 4 client.15237 mds_command: new command op to 24478 tid=0[{"prefix": "status"}] 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 --> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56061fa53e60 con 0x56061fb7c0b0 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.762+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb7c0b0 0x56061fb78b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaba59c700 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb7c0b0 0x56061fb78b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ec9668065ab92a32 server_cookie=1cdbe966e0018c07 in_seq=0 out_seq=0 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaa1ffb700 10 client.15237 ms_handle_connect on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaa1ffb700 1 -- 172.21.15.64:0/2966736253 <== mds.4 v2:172.21.15.144:6834/4223527896 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56061fa53e60 con 0x56061fb7c0b0 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaa1ffb700 10 client.15237 handle_command_reply: tid=0 2023-07-14T18:24:30.825 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 client.15237 shutdown 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 msgr2=0x56061fb76e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ffab40a1800 tx=0x7ffab40098a0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 shutdown_connections 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x56061fb7c0b0 0x56061fb78b40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56061fb0b460 0x56061fb76e30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x56061fb09f10 0x56061fb04aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.826 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 --2- 172.21.15.64:0/2966736253 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x56061fb0a860 0x56061fb74860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaa1ffb700 0 client.15237 ms_handle_reset on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 >> 172.21.15.64:0/2966736253 conn(0x56061faff630 msgr2=0x56061fb0e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 shutdown_connections 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 1 -- 172.21.15.64:0/2966736253 wait complete. 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.763+0000 7ffaccda6b80 20 client.15237 trim_cache size 0 max 16384 2023-07-14T18:24:30.827 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 4, 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24478, 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:30.828 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 63.504028761000001, 2023-07-14T18:24:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:30.829 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 105.041548783 2023-07-14T18:24:30.829 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:30.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 msgr2=0x7ffaa80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 0x7ffaa80a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ffaac006010 tx=0x7ffaac00b260 comp rx=0 tx=0).stop 2023-07-14T18:24:30.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 msgr2=0x7ffabc19ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eb80 tx=0x7ffab400ef40 comp rx=0 tx=0).stop 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 shutdown_connections 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ffaa80a11e0 0x7ffaa80a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffabc105900 0x7ffabc19ce20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ffabc1006c0 0x7ffabc195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 --2- 172.21.15.64:0/2408798333 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ffabc102fe0 0x7ffabc195db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.831 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 >> 172.21.15.64:0/2408798333 conn(0x7ffabc0fa160 msgr2=0x7ffabc101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 shutdown_connections 2023-07-14T18:24:30.832 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.764+0000 7ffac0c0e700 1 -- 172.21.15.64:0/2408798333 wait complete. 2023-07-14T18:24:30.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T18:24:30.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 -- 172.21.15.64:0/2742163392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f80ffd00 msgr2=0x7f91f8100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 --2- 172.21.15.64:0/2742163392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f80ffd00 0x7f91f8100150 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009990 tx=0x7f91e8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 -- 172.21.15.64:0/2742163392 shutdown_connections 2023-07-14T18:24:30.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 --2- 172.21.15.64:0/2742163392 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f8100690 0x7f91f8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.960 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 --2- 172.21.15.64:0/2742163392 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 0x7f91f80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 --2- 172.21.15.64:0/2742163392 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f80ffd00 0x7f91f8100150 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 -- 172.21.15.64:0/2742163392 >> 172.21.15.64:0/2742163392 conn(0x7f91f80fa130 msgr2=0x7f91f80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 -- 172.21.15.64:0/2742163392 shutdown_connections 2023-07-14T18:24:30.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 -- 172.21.15.64:0/2742163392 wait complete. 2023-07-14T18:24:30.961 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.907+0000 7f91fe051700 1 Processor -- start 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 -- start start 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 0x7f91f8068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f80ffd00 0x7f91f819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f819d440 con 0x7f91f8100690 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f819d5b0 con 0x7f91f80feb10 2023-07-14T18:24:30.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91fe051700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f819d770 con 0x7f91f80ffd00 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48212/0 (socket says 172.21.15.64:48212) 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 -- 172.21.15.64:0/25612299 learned_addr learned my addr 172.21.15.64:0/25612299 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f80ffd00 0x7f91f819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f77fe700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 0x7f91f8068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 -- 172.21.15.64:0/25612299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f80ffd00 msgr2=0x7f91f819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f80ffd00 0x7f91f819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 -- 172.21.15.64:0/25612299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 msgr2=0x7f91f8068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 0x7f91f8068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.908+0000 7f91f7fff700 1 -- 172.21.15.64:0/25612299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80095f0 con 0x7f91f8100690 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91f7fff700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f91f0007ee0 tx=0x7f91f000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91f4ff9700 1 -- 172.21.15.64:0/25612299 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0017070 con 0x7f91f8100690 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91f4ff9700 1 -- 172.21.15.64:0/25612299 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f91f000f040 con 0x7f91f8100690 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f819da50 con 0x7f91f8100690 2023-07-14T18:24:30.965 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91f4ff9700 1 -- 172.21.15.64:0/25612299 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f0005710 con 0x7f91f8100690 2023-07-14T18:24:30.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.909+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f819df60 con 0x7f91f8100690 2023-07-14T18:24:30.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.911+0000 7f91f4ff9700 1 -- 172.21.15.64:0/25612299 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f91f0005870 con 0x7f91f8100690 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.911+0000 7f91f4ff9700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 0x7f91e40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.912+0000 7f91f77fe700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 0x7f91e40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.912+0000 7f91f4ff9700 1 -- 172.21.15.64:0/25612299 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f91f00cc570 con 0x7f91f8100690 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.912+0000 7f91f77fe700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 0x7f91e40a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f91f80ffb60 tx=0x7f91e000b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.912+0000 7f920a1e9b80 1 Processor -- start 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.912+0000 7f920a1e9b80 1 -- start start 2023-07-14T18:24:30.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce00e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce018d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcd04c60 con 0x563ddcdfff10 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcd579f0 con 0x563ddce01460 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcd50e80 con 0x563ddce00860 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48216/0 (socket says 172.21.15.64:48216) 2023-07-14T18:24:30.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 -- 172.21.15.64:0/478909047 learned_addr learned my addr 172.21.15.64:0/478909047 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f7fff700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce00e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 -- 172.21.15.64:0/478909047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 msgr2=0x563ddce00e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce00e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 -- 172.21.15.64:0/478909047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 msgr2=0x563ddce018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 -- 172.21.15.64:0/478909047 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ddcb7ed90 con 0x563ddcdfff10 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f7fff700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91f77fe700 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f91e000fcf0 tx=0x7f91e0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a8b07f5514fdb2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91edffb700 1 -- 172.21.15.64:0/478909047 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0011b80 con 0x563ddcdfff10 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91edffb700 1 -- 172.21.15.64:0/478909047 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f91e0011ce0 con 0x563ddcdfff10 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.913+0000 7f91edffb700 1 -- 172.21.15.64:0/478909047 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e001d610 con 0x563ddcdfff10 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- 172.21.15.64:0/478909047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 msgr2=0x563ddce00320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f91e000fcf0 tx=0x7f91e0012000 comp rx=0 tx=0).stop 2023-07-14T18:24:30.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- 172.21.15.64:0/478909047 shutdown_connections 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce00320 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/478909047 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce00e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- 172.21.15.64:0/478909047 >> 172.21.15.64:0/478909047 conn(0x563ddcdf5630 msgr2=0x563ddcdfa980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- 172.21.15.64:0/478909047 shutdown_connections 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- 172.21.15.64:0/478909047 wait complete. 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.914+0000 7f920a1e9b80 1 -- start start 2023-07-14T18:24:30.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 Processor -- start 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddcdfff10 0x563ddce098e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce00860 0x563ddce09e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddccf0a70 con 0x563ddce01460 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddccf0770 con 0x563ddce00860 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcbd48d0 con 0x563ddcdfff10 2023-07-14T18:24:30.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f920a1e9b80 10 client.0 _ll_register_callbacks cb 0x563ddccee710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f91f7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f91f7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48226/0 (socket says 172.21.15.64:48226) 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f91f7fff700 1 -- 172.21.15.64:0/3111473526 learned_addr learned my addr 172.21.15.64:0/3111473526 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce00860 0x563ddce09e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.915+0000 7f91f77fe700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddcdfff10 0x563ddce098e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 -- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddcdfff10 msgr2=0x563ddce098e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddcdfff10 0x563ddce098e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 -- 172.21.15.64:0/3111473526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce00860 msgr2=0x563ddce09e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce00860 0x563ddce09e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 -- 172.21.15.64:0/3111473526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e000f9a0 con 0x563ddce01460 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91f7fff700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f91f00991b0 tx=0x7f91f0099120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17310e16ce643117 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91f00994b0 con 0x563ddce01460 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f91f009d4e0 con 0x563ddce01460 2023-07-14T18:24:30.977 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563ddccaa4a0 con 0x563ddce01460 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91f00a1900 con 0x563ddce01460 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f920a1e9b80 10 client.15249 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563ddcb67c10 con 0x563ddce01460 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f91f00b5080 con 0x563ddce01460 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.916+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91f009d160 con 0x563ddce01460 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 10 client.15249 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 10 client.15249 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T18:24:30.978 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 20 client.15249 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:30.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce14b30 0x563ddce13fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 4 client.15249 mds_command: new command op to 24487 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:30.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 --> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563ddcdced50 con 0x563ddce14b30 2023-07-14T18:24:30.979 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f91f77fe700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce14b30 0x563ddce13fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f91d6ffd700 10 client.15249 ms_handle_connect on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:30.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.917+0000 7f91f77fe700 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce14b30 0x563ddce13fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=56faf6ec90e7c5ae server_cookie=c9f18f097c0894e1 in_seq=0 out_seq=0 2023-07-14T18:24:30.980 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f91d6ffd700 1 -- 172.21.15.64:0/3111473526 <== mds.3 v2:172.21.15.144:6836/1582348642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x563ddcdced50 con 0x563ddce14b30 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f91d6ffd700 10 client.15249 handle_command_reply: tid=0 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 client.15249 shutdown 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 msgr2=0x563ddce10e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f91f00991b0 tx=0x7f91f0099120 comp rx=0 tx=0).stop 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 shutdown_connections 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce14b30 0x563ddce13fc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddcdfff10 0x563ddce098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.981 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f91d6ffd700 0 client.15249 ms_handle_reset on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce01460 0x563ddce10e60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/3111473526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce00860 0x563ddce09e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 >> 172.21.15.64:0/3111473526 conn(0x563ddcdf5630 msgr2=0x563ddcdf9600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 shutdown_connections 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.918+0000 7f920a1e9b80 1 -- 172.21.15.64:0/3111473526 wait complete. 2023-07-14T18:24:30.982 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.919+0000 7f920a1e9b80 20 client.15249 trim_cache size 0 max 16384 2023-07-14T18:24:30.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 Processor -- start 2023-07-14T18:24:30.989 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 -- start start 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce10e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddce14850 con 0x563ddcdfff10 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcdced50 con 0x563ddce01460 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddcd1f790 con 0x563ddce00860 2023-07-14T18:24:30.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48230/0 (socket says 172.21.15.64:48230) 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 -- 172.21.15.64:0/1820486590 learned_addr learned my addr 172.21.15.64:0/1820486590 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 -- 172.21.15.64:0/1820486590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 msgr2=0x563ddce047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f7fff700 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce10e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 -- 172.21.15.64:0/1820486590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 msgr2=0x563ddce10e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce10e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.936+0000 7f91f77fe700 1 -- 172.21.15.64:0/1820486590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563ddcba5200 con 0x563ddcdfff10 2023-07-14T18:24:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f91e00117d0 tx=0x7f91e0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfa91c21907164a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f91ee7fc700 1 -- 172.21.15.64:0/1820486590 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0011900 con 0x563ddcdfff10 2023-07-14T18:24:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f91ee7fc700 1 -- 172.21.15.64:0/1820486590 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f91e002e040 con 0x563ddcdfff10 2023-07-14T18:24:30.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f91ee7fc700 1 -- 172.21.15.64:0/1820486590 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e0005440 con 0x563ddcdfff10 2023-07-14T18:24:30.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1820486590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 msgr2=0x563ddce042a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f91e00117d0 tx=0x7f91e0015000 comp rx=0 tx=0).stop 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1820486590 shutdown_connections 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddce01460 0x563ddce10e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddcdfff10 0x563ddce042a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1820486590 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce00860 0x563ddce047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:30.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1820486590 >> 172.21.15.64:0/1820486590 conn(0x563ddcdf5630 msgr2=0x563ddce07750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1820486590 shutdown_connections 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.937+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1820486590 wait complete. 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 -- start start 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 Processor -- start 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddcdfff10 0x563ddce0ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce01460 0x563ddce0f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddce0fdd0 con 0x563ddce00860 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddce0ff40 con 0x563ddcdfff10 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x563ddce10100 con 0x563ddce01460 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f920a1e9b80 10 client.0 _ll_register_callbacks cb 0x7f91f00958a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48238/0 (socket says 172.21.15.64:48238) 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 -- 172.21.15.64:0/1884366893 learned_addr learned my addr 172.21.15.64:0/1884366893 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:30.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddcdfff10 0x563ddce0ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f7fff700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce01460 0x563ddce0f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:30.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 -- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce01460 msgr2=0x563ddce0f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:30.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce01460 0x563ddce0f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 -- 172.21.15.64:0/1884366893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddcdfff10 msgr2=0x563ddce0ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.938+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddcdfff10 0x563ddce0ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91f6ffd700 1 -- 172.21.15.64:0/1884366893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563ddcb7ed90 con 0x563ddce00860 2023-07-14T18:24:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91f6ffd700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098e0 tx=0x7f91e8005320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0221b185dd47d1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e801d070 con 0x563ddce00860 2023-07-14T18:24:31.001 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f91e8004500 con 0x563ddce00860 2023-07-14T18:24:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91e0011ce0 con 0x563ddce00860 2023-07-14T18:24:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91e8009ee0 con 0x563ddce00860 2023-07-14T18:24:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 10 client.15255 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.002 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x563ddcb7d260 con 0x563ddce00860 2023-07-14T18:24:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f91e0011ce0 con 0x563ddce00860 2023-07-14T18:24:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f91e8003a50 con 0x563ddce00860 2023-07-14T18:24:31.003 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 10 client.15255 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 10 client.15255 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T18:24:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 20 client.15255 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.004 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce726b0 0x563ddce6f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 4 client.15255 mds_command: new command op to 24487 tid=0[{"prefix": "status"}] 2023-07-14T18:24:31.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.939+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 --> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563ddcd49e60 con 0x563ddce726b0 2023-07-14T18:24:31.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce726b0 0x563ddce6f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.005 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f91d6ffd700 10 client.15255 ms_handle_connect on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f91f77fe700 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce726b0 0x563ddce6f020 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=957dc85e13e66a0e server_cookie=c5d4749de719a970 in_seq=0 out_seq=0 2023-07-14T18:24:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f91d6ffd700 1 -- 172.21.15.64:0/1884366893 <== mds.3 v2:172.21.15.144:6836/1582348642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x563ddcd49e60 con 0x563ddce726b0 2023-07-14T18:24:31.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f91d6ffd700 10 client.15255 handle_command_reply: tid=0 2023-07-14T18:24:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f920a1e9b80 1 client.15255 shutdown 2023-07-14T18:24:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 msgr2=0x563ddce0f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.940+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f91e80098e0 tx=0x7f91e8005320 comp rx=0 tx=0).stop 2023-07-14T18:24:31.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 shutdown_connections 2023-07-14T18:24:31.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x563ddce726b0 0x563ddce6f020 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x563ddce01460 0x563ddce0f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x563ddcdfff10 0x563ddce0ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91d6ffd700 0 client.15255 ms_handle_reset on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:31.008 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 --2- 172.21.15.64:0/1884366893 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563ddce00860 0x563ddce0f350 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 >> 172.21.15.64:0/1884366893 conn(0x563ddcdf5630 msgr2=0x563ddce042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 shutdown_connections 2023-07-14T18:24:31.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 1 -- 172.21.15.64:0/1884366893 wait complete. 2023-07-14T18:24:31.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f920a1e9b80 20 client.15255 trim_cache size 0 max 16384 2023-07-14T18:24:31.010 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:31.010 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 3, 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24487, 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 64.688001392999993, 2023-07-14T18:24:31.011 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:31.012 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 102.837958151 2023-07-14T18:24:31.012 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:31.013 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 msgr2=0x7f91e40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.013 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 0x7f91e40a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f91f80ffb60 tx=0x7f91e000b5b0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 msgr2=0x7f91f819cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f91f0007ee0 tx=0x7f91f000d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 shutdown_connections 2023-07-14T18:24:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f91e40a11e0 0x7f91e40a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.014 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f91f8100690 0x7f91f819cca0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91f80feb10 0x7f91f8068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 --2- 172.21.15.64:0/25612299 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f91f80ffd00 0x7f91f819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 >> 172.21.15.64:0/25612299 conn(0x7f91f80fa130 msgr2=0x7f91f80fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 shutdown_connections 2023-07-14T18:24:31.015 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:30.941+0000 7f91fe051700 1 -- 172.21.15.64:0/25612299 wait complete. 2023-07-14T18:24:31.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T18:24:31.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 -- 172.21.15.64:0/2026710428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e41017c0 msgr2=0x7fa4e4108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.144 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/2026710428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e41017c0 0x7fa4e4108080 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc00b0a0 tx=0x7fa4dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 -- 172.21.15.64:0/2026710428 shutdown_connections 2023-07-14T18:24:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/2026710428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e41017c0 0x7fa4e4108080 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/2026710428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e40ffc40 0x7fa4e4100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/2026710428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e4100e30 0x7fa4e4101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.089+0000 7fa4eabaa700 1 -- 172.21.15.64:0/2026710428 >> 172.21.15.64:0/2026710428 conn(0x7fa4e40fb260 msgr2=0x7fa4e40fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- 172.21.15.64:0/2026710428 shutdown_connections 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- 172.21.15.64:0/2026710428 wait complete. 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 Processor -- start 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- start start 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e40ffc40 0x7fa4e4195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 0x7fa4e419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e419d4c0 con 0x7fa4e4100e30 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e419d600 con 0x7fa4e41017c0 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4eabaa700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e419d790 con 0x7fa4e40ffc40 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.090+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48252/0 (socket says 172.21.15.64:48252) 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 -- 172.21.15.64:0/3507517580 learned_addr learned my addr 172.21.15.64:0/3507517580 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e8946700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e40ffc40 0x7fa4e4195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e9147700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 0x7fa4e419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 -- 172.21.15.64:0/3507517580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e40ffc40 msgr2=0x7fa4e4195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e40ffc40 0x7fa4e4195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 -- 172.21.15.64:0/3507517580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 msgr2=0x7fa4e419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 0x7fa4e419cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 -- 172.21.15.64:0/3507517580 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc009d00 con 0x7fa4e4100e30 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e9147700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 0x7fa4e419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c9b0 tx=0x7fa4d000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e1ffb700 1 -- 172.21.15.64:0/3507517580 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0007b00 con 0x7fa4e4100e30 2023-07-14T18:24:31.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e1ffb700 1 -- 172.21.15.64:0/3507517580 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa4d0007c60 con 0x7fa4e4100e30 2023-07-14T18:24:31.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e419da70 con 0x7fa4e4100e30 2023-07-14T18:24:31.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4e1ffb700 1 -- 172.21.15.64:0/3507517580 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0018660 con 0x7fa4e4100e30 2023-07-14T18:24:31.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.091+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e419df80 con 0x7fa4e4100e30 2023-07-14T18:24:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.093+0000 7fa4e1ffb700 1 -- 172.21.15.64:0/3507517580 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa4d000f450 con 0x7fa4e4100e30 2023-07-14T18:24:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.094+0000 7fa4e1ffb700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 0x7fa4d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.095+0000 7fa4e8946700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 0x7fa4d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.095+0000 7fa4e1ffb700 1 -- 172.21.15.64:0/3507517580 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4d00ccd90 con 0x7fa4e4100e30 2023-07-14T18:24:31.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.095+0000 7fa4e8946700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 0x7fa4d40a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4100c90 tx=0x7fa4d800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 Processor -- start 2023-07-14T18:24:31.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 -- start start 2023-07-14T18:24:31.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df9be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741df9c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741de9fc60 con 0x55741df9af10 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741def29f0 con 0x55741df9c460 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741deebe80 con 0x55741df9b860 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48258/0 (socket says 172.21.15.64:48258) 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 -- 172.21.15.64:0/1524238799 learned_addr learned my addr 172.21.15.64:0/1524238799 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df9be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e9147700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741df9c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 -- 172.21.15.64:0/1524238799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 msgr2=0x55741df9be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df9be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 -- 172.21.15.64:0/1524238799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 msgr2=0x55741df9c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741df9c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e8946700 1 -- 172.21.15.64:0/1524238799 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55741dd19d90 con 0x55741df9af10 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.096+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df9be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:31.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4e8946700 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800fd80 tx=0x7fa4d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bec04db60b5dab1c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4ceffd700 1 -- 172.21.15.64:0/1524238799 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8010ac0 con 0x55741df9af10 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4ceffd700 1 -- 172.21.15.64:0/1524238799 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa4d8020070 con 0x55741df9af10 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4ceffd700 1 -- 172.21.15.64:0/1524238799 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d801bd40 con 0x55741df9af10 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/1524238799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 msgr2=0x55741df9b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fa4d800fd80 tx=0x7fa4d8012000 comp rx=0 tx=0).stop 2023-07-14T18:24:31.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/1524238799 shutdown_connections 2023-07-14T18:24:31.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741df9c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df9b320 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/1524238799 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df9be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/1524238799 >> 172.21.15.64:0/1524238799 conn(0x55741df90630 msgr2=0x55741df95980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.097+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/1524238799 shutdown_connections 2023-07-14T18:24:31.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/1524238799 wait complete. 2023-07-14T18:24:31.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 -- start start 2023-07-14T18:24:31.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 Processor -- start 2023-07-14T18:24:31.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741dfabed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741de8ba70 con 0x55741df9b860 2023-07-14T18:24:31.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741de8b770 con 0x55741df9af10 2023-07-14T18:24:31.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741dd6f8d0 con 0x55741df9c460 2023-07-14T18:24:31.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4f6d42b80 10 client.0 _ll_register_callbacks cb 0x55741de89710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.163 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48266/0 (socket says 172.21.15.64:48266) 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.098+0000 7fa4e3fff700 1 -- 172.21.15.64:0/91351560 learned_addr learned my addr 172.21.15.64:0/91351560 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e9147700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741dfabed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e8946700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 -- 172.21.15.64:0/91351560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 msgr2=0x55741dfabed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741dfabed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.164 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 -- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 msgr2=0x55741dfa4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 -- 172.21.15.64:0/91351560 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d800fa60 con 0x55741df9b860 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000c00 tx=0x7fa4d0099980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d9ba10fb2a4a3b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d009dd60 con 0x55741df9b860 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4d009c030 con 0x55741df9b860 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55741de454a0 con 0x55741df9b860 2023-07-14T18:24:31.165 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4d00a2180 con 0x55741df9b860 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4f6d42b80 10 client.15267 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55741dd02c10 con 0x55741df9b860 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa4d00ca080 con 0x55741df9b860 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.099+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa4d0099300 con 0x55741df9b860 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 10 client.15267 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 10 client.15267 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T18:24:31.166 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 20 client.15267 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741dfaffd0 0x55741dfaee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 4 client.15267 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 --> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55741df69d50 con 0x55741dfaffd0 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4e8946700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741dfaffd0 0x55741dfaee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4c3fff700 10 client.15267 ms_handle_connect on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.100+0000 7fa4e8946700 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741dfaffd0 0x55741dfaee30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b5aea3e2e25b502b server_cookie=6f4cad1068bb32ae in_seq=0 out_seq=0 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4c3fff700 1 -- 172.21.15.64:0/91351560 <== mds.2 v2:172.21.15.105:6836/2898950423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55741df69d50 con 0x55741dfaffd0 2023-07-14T18:24:31.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4c3fff700 10 client.15267 handle_command_reply: tid=0 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 client.15267 shutdown 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 msgr2=0x55741dfa4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fa4d0000c00 tx=0x7fa4d0099980 comp rx=0 tx=0).stop 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 shutdown_connections 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741dfaffd0 0x55741dfaee30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741dfabed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/91351560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfa4e90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4c3fff700 0 client.15267 ms_handle_reset on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 >> 172.21.15.64:0/91351560 conn(0x55741df90630 msgr2=0x55741df94600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 shutdown_connections 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.101+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/91351560 wait complete. 2023-07-14T18:24:31.169 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.102+0000 7fa4f6d42b80 20 client.15267 trim_cache size 0 max 16384 2023-07-14T18:24:31.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 Processor -- start 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 -- start start 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741df9f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741df90250 con 0x55741df9b860 2023-07-14T18:24:31.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741df69d50 con 0x55741df9af10 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741dd7bde0 con 0x55741df9c460 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4e3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48274/0 (socket says 172.21.15.64:48274) 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.119+0000 7fa4e3fff700 1 -- 172.21.15.64:0/2370713983 learned_addr learned my addr 172.21.15.64:0/2370713983 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e8946700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e9147700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741df9f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.174 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 -- 172.21.15.64:0/2370713983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 msgr2=0x55741df9f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741df9f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 -- 172.21.15.64:0/2370713983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 msgr2=0x55741dfa2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 -- 172.21.15.64:0/2370713983 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55741dd40200 con 0x55741df9b860 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e9147700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741df9f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00074a0 tx=0x7fa4d0099eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42ec55b2c82c6857 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4cf7fe700 1 -- 172.21.15.64:0/2370713983 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d00a1f80 con 0x55741df9b860 2023-07-14T18:24:31.175 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4cf7fe700 1 -- 172.21.15.64:0/2370713983 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55741de454a0 con 0x55741df9b860 2023-07-14T18:24:31.176 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.120+0000 7fa4cf7fe700 1 -- 172.21.15.64:0/2370713983 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d000f820 con 0x55741df9b860 2023-07-14T18:24:31.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/2370713983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 msgr2=0x55741dfaee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa4d00074a0 tx=0x7fa4d0099eb0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/2370713983 shutdown_connections 2023-07-14T18:24:31.179 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9c460 0x55741df9f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9af10 0x55741dfa2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/2370713983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9b860 0x55741dfaee30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/2370713983 >> 172.21.15.64:0/2370713983 conn(0x55741df90630 msgr2=0x55741dfa3650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/2370713983 shutdown_connections 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/2370713983 wait complete. 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- start start 2023-07-14T18:24:31.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 Processor -- start 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df95870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741e007660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741e007c20 con 0x55741df9af10 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741e007d90 con 0x55741df9c460 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.121+0000 7fa4f6d42b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55741e007f50 con 0x55741df9b860 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4f6d42b80 10 client.0 _ll_register_callbacks cb 0x55741de89710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.181 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48278/0 (socket says 172.21.15.64:48278) 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 -- 172.21.15.64:0/860995578 learned_addr learned my addr 172.21.15.64:0/860995578 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df95870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e9147700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741e007660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 -- 172.21.15.64:0/860995578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 msgr2=0x55741df95870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df95870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 -- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 msgr2=0x55741e007660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741e007660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 -- 172.21.15.64:0/860995578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55741dd02c10 con 0x55741df9af10 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e9147700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741e007660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4e8946700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005e40 tx=0x7fa4d80318c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e389667b152c4c16 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4c3fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8031070 con 0x55741df9af10 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4d8036030 con 0x55741df9af10 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4d8020070 con 0x55741df9af10 2023-07-14T18:24:31.183 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4d80311d0 con 0x55741df9af10 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4f6d42b80 10 client.15273 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.122+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55741dd18260 con 0x55741df9af10 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa4d8020070 con 0x55741df9af10 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa4d8005200 con 0x55741df9af10 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 10 client.15273 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 10 client.15273 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T18:24:31.184 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 20 client.15273 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741e00cbf0 0x55741e009540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 4 client.15273 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 --> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55741dee4e60 con 0x55741e00cbf0 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741e00cbf0 0x55741e009540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4c3fff700 10 client.15273 ms_handle_connect on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.123+0000 7fa4e3fff700 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741e00cbf0 0x55741e009540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d880f234113a0d99 server_cookie=48f5b0d2d08225c8 in_seq=0 out_seq=0 2023-07-14T18:24:31.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4c3fff700 1 -- 172.21.15.64:0/860995578 <== mds.2 v2:172.21.15.105:6836/2898950423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55741dee4e60 con 0x55741e00cbf0 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4c3fff700 10 client.15273 handle_command_reply: tid=0 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 client.15273 shutdown 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 msgr2=0x55741df95330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8005e40 tx=0x7fa4d80318c0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 shutdown_connections 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x55741e00cbf0 0x55741e009540 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55741df9c460 0x55741e007660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55741df9af10 0x55741df95330 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4c3fff700 0 client.15273 ms_handle_reset on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 --2- 172.21.15.64:0/860995578 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55741df9b860 0x55741df95870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 >> 172.21.15.64:0/860995578 conn(0x55741df90630 msgr2=0x55741dfaee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 shutdown_connections 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 1 -- 172.21.15.64:0/860995578 wait complete. 2023-07-14T18:24:31.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4f6d42b80 20 client.15273 trim_cache size 0 max 16384 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24500, 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:31.188 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 65.882901680000003, 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 100.84195511 2023-07-14T18:24:31.189 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:31.190 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 msgr2=0x7fa4d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 0x7fa4d40a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa4e4100c90 tx=0x7fa4d800a380 comp rx=0 tx=0).stop 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 msgr2=0x7fa4e4195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.124+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fa4d000c9b0 tx=0x7fa4d000ccc0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 shutdown_connections 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fa4e40ffc40 0x7fa4e4195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa4d40a11e0 0x7fa4d40a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.191 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa4e41017c0 0x7fa4e419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 --2- 172.21.15.64:0/3507517580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4e4100e30 0x7fa4e4195e80 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 >> 172.21.15.64:0/3507517580 conn(0x7fa4e40fb260 msgr2=0x7fa4e4104290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 shutdown_connections 2023-07-14T18:24:31.192 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.125+0000 7fa4eabaa700 1 -- 172.21.15.64:0/3507517580 wait complete. 2023-07-14T18:24:31.192 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T18:24:31.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- 172.21.15.64:0/3978246037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 msgr2=0x7f4e40106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 --2- 172.21.15.64:0/3978246037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e40106f50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f4e3800b0a0 tx=0x7f4e3800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- 172.21.15.64:0/3978246037 shutdown_connections 2023-07-14T18:24:31.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 --2- 172.21.15.64:0/3978246037 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 0x7f4e400fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 --2- 172.21.15.64:0/3978246037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e40106f50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 --2- 172.21.15.64:0/3978246037 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- 172.21.15.64:0/3978246037 >> 172.21.15.64:0/3978246037 conn(0x7f4e400fa130 msgr2=0x7f4e400fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- 172.21.15.64:0/3978246037 shutdown_connections 2023-07-14T18:24:31.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- 172.21.15.64:0/3978246037 wait complete. 2023-07-14T18:24:31.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 Processor -- start 2023-07-14T18:24:31.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.267+0000 7f4e470c3700 1 -- start start 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 0x7f4e401948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4019c420 con 0x7f4e40100690 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4019c560 con 0x7f4e400ffd00 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e470c3700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4019c6f0 con 0x7f4e400feb10 2023-07-14T18:24:31.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48296/0 (socket says 172.21.15.64:48296) 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 -- 172.21.15.64:0/101551137 learned_addr learned my addr 172.21.15.64:0/101551137 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 0x7f4e401948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 -- 172.21.15.64:0/101551137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 msgr2=0x7f4e401948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 0x7f4e401948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.326 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 -- 172.21.15.64:0/101551137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 msgr2=0x7f4e40194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 -- 172.21.15.64:0/101551137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e38009d00 con 0x7f4e40100690 2023-07-14T18:24:31.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e45660700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e380095e0 tx=0x7f4e3800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.327 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.268+0000 7f4e3dffb700 1 -- 172.21.15.64:0/101551137 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3800e040 con 0x7f4e40100690 2023-07-14T18:24:31.328 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.269+0000 7f4e3dffb700 1 -- 172.21.15.64:0/101551137 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e38003c00 con 0x7f4e40100690 2023-07-14T18:24:31.328 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.269+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4019c970 con 0x7f4e40100690 2023-07-14T18:24:31.328 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.269+0000 7f4e3dffb700 1 -- 172.21.15.64:0/101551137 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e380052a0 con 0x7f4e40100690 2023-07-14T18:24:31.328 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.269+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4019ce40 con 0x7f4e40100690 2023-07-14T18:24:31.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.271+0000 7f4e3dffb700 1 -- 172.21.15.64:0/101551137 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4e3801d470 con 0x7f4e40100690 2023-07-14T18:24:31.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e3dffb700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 0x7f4e300a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e3dffb700 1 -- 172.21.15.64:0/101551137 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e380cdfd0 con 0x7f4e40100690 2023-07-14T18:24:31.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 0x7f4e300a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 0x7f4e300a3710 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4e400ffb60 tx=0x7f4e34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e5325bb80 1 Processor -- start 2023-07-14T18:24:31.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.272+0000 7f4e5325bb80 1 -- start start 2023-07-14T18:24:31.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21db2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21cb6c60 con 0x55fe21db3460 2023-07-14T18:24:31.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21d099f0 con 0x55fe21db1f10 2023-07-14T18:24:31.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21d02e80 con 0x55fe21db2860 2023-07-14T18:24:31.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48304/0 (socket says 172.21.15.64:48304) 2023-07-14T18:24:31.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 -- 172.21.15.64:0/3808556525 learned_addr learned my addr 172.21.15.64:0/3808556525 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21db2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 -- 172.21.15.64:0/3808556525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 msgr2=0x55fe21db2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21db2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 -- 172.21.15.64:0/3808556525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 msgr2=0x55fe21db2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 -- 172.21.15.64:0/3808556525 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe21b30d90 con 0x55fe21db3460 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21db2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e45660700 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4e38097500 tx=0x7f4e3809ac10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff8e8f1bd279ed0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.273+0000 7f4e2affd700 1 -- 172.21.15.64:0/3808556525 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3809f030 con 0x55fe21db3460 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e2affd700 1 -- 172.21.15.64:0/3808556525 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e380a33c0 con 0x55fe21db3460 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e2affd700 1 -- 172.21.15.64:0/3808556525 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e380978e0 con 0x55fe21db3460 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3808556525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 msgr2=0x55fe21db38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f4e38097500 tx=0x7f4e3809ac10 comp rx=0 tx=0).stop 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3808556525 shutdown_connections 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21db38d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3808556525 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21db2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3808556525 >> 172.21.15.64:0/3808556525 conn(0x55fe21da7630 msgr2=0x55fe21dac980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3808556525 shutdown_connections 2023-07-14T18:24:31.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.274+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3808556525 wait complete. 2023-07-14T18:24:31.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 -- start start 2023-07-14T18:24:31.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 Processor -- start 2023-07-14T18:24:31.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21dbb500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21dbba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21ca2a70 con 0x55fe21db3460 2023-07-14T18:24:31.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21ca2770 con 0x55fe21db1f10 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21b868d0 con 0x55fe21db2860 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e5325bb80 10 client.0 _ll_register_callbacks cb 0x55fe21ca0710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e45660700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48310/0 (socket says 172.21.15.64:48310) 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.275+0000 7f4e45660700 1 -- 172.21.15.64:0/2321292323 learned_addr learned my addr 172.21.15.64:0/2321292323 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21dbb500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21dbba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 -- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 msgr2=0x55fe21dbba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21dbba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 -- 172.21.15.64:0/2321292323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 msgr2=0x55fe21dbb500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21dbb500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 -- 172.21.15.64:0/2321292323 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4019ce40 con 0x55fe21db3460 2023-07-14T18:24:31.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e45660700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4e38099350 tx=0x7f4e3809b180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11a53aad26c9d0c2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3809b470 con 0x55fe21db3460 2023-07-14T18:24:31.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e3809a4f0 con 0x55fe21db3460 2023-07-14T18:24:31.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe21c5c4a0 con 0x55fe21db3460 2023-07-14T18:24:31.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e3809a6b0 con 0x55fe21db3460 2023-07-14T18:24:31.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e5325bb80 10 client.15285 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe21b19c10 con 0x55fe21db3460 2023-07-14T18:24:31.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.276+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e380cb020 con 0x55fe21db3460 2023-07-14T18:24:31.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e380a2f30 con 0x55fe21db3460 2023-07-14T18:24:31.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 10 client.15285 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 10 client.15285 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:24:31.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 20 client.15285 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21dc6730 0x55fe21dc59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 4 client.15285 mds_command: new command op to 24508 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:31.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 --> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe21d80d50 con 0x55fe21dc6730 2023-07-14T18:24:31.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21dc6730 0x55fe21dc59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21dc6730 0x55fe21dc59b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0697a5e381edf70 server_cookie=46e998280863c2e6 in_seq=0 out_seq=0 2023-07-14T18:24:31.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.277+0000 7f4e1ffff700 10 client.15285 ms_handle_connect on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:31.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e1ffff700 1 -- 172.21.15.64:0/2321292323 <== mds.1 v2:172.21.15.144:6838/4051776603 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55fe21d80d50 con 0x55fe21dc6730 2023-07-14T18:24:31.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e1ffff700 10 client.15285 handle_command_reply: tid=0 2023-07-14T18:24:31.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 client.15285 shutdown 2023-07-14T18:24:31.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 msgr2=0x55fe21dc2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4e38099350 tx=0x7f4e3809b180 comp rx=0 tx=0).stop 2023-07-14T18:24:31.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 shutdown_connections 2023-07-14T18:24:31.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21dc6730 0x55fe21dc59b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db3460 0x55fe21dc2a80 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e1ffff700 0 client.15285 ms_handle_reset on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:31.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21dbb500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/2321292323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db2860 0x55fe21dbba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 >> 172.21.15.64:0/2321292323 conn(0x55fe21da7630 msgr2=0x55fe21dab600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 shutdown_connections 2023-07-14T18:24:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 1 -- 172.21.15.64:0/2321292323 wait complete. 2023-07-14T18:24:31.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.278+0000 7f4e5325bb80 20 client.15285 trim_cache size 0 max 16384 2023-07-14T18:24:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.296+0000 7f4e5325bb80 1 Processor -- start 2023-07-14T18:24:31.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 -- start start 2023-07-14T18:24:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db3460 0x55fe21dcee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.363 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21d80d50 con 0x55fe21db2860 2023-07-14T18:24:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21dade30 con 0x55fe21db1f10 2023-07-14T18:24:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21dbc6f0 con 0x55fe21db3460 2023-07-14T18:24:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48316/0 (socket says 172.21.15.64:48316) 2023-07-14T18:24:31.364 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 -- 172.21.15.64:0/633920660 learned_addr learned my addr 172.21.15.64:0/633920660 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e45660700 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db3460 0x55fe21dcee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 -- 172.21.15.64:0/633920660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db3460 msgr2=0x55fe21dcee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db3460 0x55fe21dcee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 -- 172.21.15.64:0/633920660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 msgr2=0x55fe21db6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.365 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 -- 172.21.15.64:0/633920660 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe21b57200 con 0x55fe21db2860 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.297+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009d90 tx=0x7f4e2c013740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20535e83817740f8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e2b7fe700 1 -- 172.21.15.64:0/633920660 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c018070 con 0x55fe21db2860 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e2b7fe700 1 -- 172.21.15.64:0/633920660 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e2c0044b0 con 0x55fe21db2860 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e2b7fe700 1 -- 172.21.15.64:0/633920660 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c011440 con 0x55fe21db2860 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 -- 172.21.15.64:0/633920660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 msgr2=0x55fe21db6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009d90 tx=0x7f4e2c013740 comp rx=0 tx=0).stop 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 -- 172.21.15.64:0/633920660 shutdown_connections 2023-07-14T18:24:31.366 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db3460 0x55fe21dcee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db1f10 0x55fe21db6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/633920660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21db6780 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 -- 172.21.15.64:0/633920660 >> 172.21.15.64:0/633920660 conn(0x55fe21da7630 msgr2=0x55fe21db96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 -- 172.21.15.64:0/633920660 shutdown_connections 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.298+0000 7f4e5325bb80 1 -- 172.21.15.64:0/633920660 wait complete. 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 -- start start 2023-07-14T18:24:31.367 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 Processor -- start 2023-07-14T18:24:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 0x55fe21e17d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.368 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 0x55fe21e1c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21daccf0 con 0x55fe21db2860 2023-07-14T18:24:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21dace60 con 0x55fe21db3460 2023-07-14T18:24:31.369 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe21e1cd30 con 0x55fe21db1f10 2023-07-14T18:24:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e5325bb80 10 client.0 _ll_register_callbacks cb 0x55fe21ca0710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e3ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.370 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e3ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48322/0 (socket says 172.21.15.64:48322) 2023-07-14T18:24:31.371 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e3ffff700 1 -- 172.21.15.64:0/3628576468 learned_addr learned my addr 172.21.15.64:0/3628576468 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.371 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e45660700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 0x55fe21e1c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.371 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.299+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 0x55fe21e17d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 -- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 msgr2=0x55fe21e17d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 0x55fe21e17d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 -- 172.21.15.64:0/3628576468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 msgr2=0x55fe21e1c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 0x55fe21e1c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.372 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 -- 172.21.15.64:0/3628576468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c009ad0 con 0x55fe21db2860 2023-07-14T18:24:31.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e45660700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 0x55fe21e1c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 0x55fe21e17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e3ffff700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c000c00 tx=0x7f4e2c004380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcda454848b8bc1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.373 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e1ffff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c018050 con 0x55fe21db2860 2023-07-14T18:24:31.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e2c003680 con 0x55fe21db2860 2023-07-14T18:24:31.374 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e380a33c0 con 0x55fe21db2860 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e2c011850 con 0x55fe21db2860 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e5325bb80 10 client.15291 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fe21b2f260 con 0x55fe21db2860 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.300+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e380a33c0 con 0x55fe21db2860 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e2c026930 con 0x55fe21db2860 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 10 client.15291 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.375 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 10 client.15291 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:24:31.376 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 20 client.15291 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.376 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21e21aa0 0x55fe21e1e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.376 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 4 client.15291 mds_command: new command op to 24508 tid=0[{"prefix": "status"}] 2023-07-14T18:24:31.377 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 --> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fe21cfbe60 con 0x55fe21e21aa0 2023-07-14T18:24:31.377 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21e21aa0 0x55fe21e1e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.377 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e44e5f700 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21e21aa0 0x55fe21e1e3f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=84942744a2edbad6 server_cookie=9d92772f7f58bff in_seq=0 out_seq=0 2023-07-14T18:24:31.377 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.301+0000 7f4e1ffff700 10 client.15291 ms_handle_connect on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:31.378 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e1ffff700 1 -- 172.21.15.64:0/3628576468 <== mds.1 v2:172.21.15.144:6838/4051776603 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55fe21cfbe60 con 0x55fe21e21aa0 2023-07-14T18:24:31.378 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e1ffff700 10 client.15291 handle_command_reply: tid=0 2023-07-14T18:24:31.378 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 client.15291 shutdown 2023-07-14T18:24:31.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 msgr2=0x55fe21e1a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c000c00 tx=0x7f4e2c004380 comp rx=0 tx=0).stop 2023-07-14T18:24:31.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 shutdown_connections 2023-07-14T18:24:31.379 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55fe21e21aa0 0x55fe21e1e3f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55fe21db1f10 0x55fe21e17d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55fe21db3460 0x55fe21e1c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.380 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e1ffff700 0 client.15291 ms_handle_reset on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 --2- 172.21.15.64:0/3628576468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe21db2860 0x55fe21e1a2a0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 >> 172.21.15.64:0/3628576468 conn(0x55fe21da7630 msgr2=0x55fe21db96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 shutdown_connections 2023-07-14T18:24:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 1 -- 172.21.15.64:0/3628576468 wait complete. 2023-07-14T18:24:31.381 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e5325bb80 20 client.15291 trim_cache size 0 max 16384 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24508, 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:31.382 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 67.070628267000004, 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 98.706251244000001 2023-07-14T18:24:31.383 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:31.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.302+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 msgr2=0x7f4e300a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 0x7f4e300a3710 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4e400ffb60 tx=0x7f4e34009260 comp rx=0 tx=0).stop 2023-07-14T18:24:31.385 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 msgr2=0x7f4e4019be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4e380095e0 tx=0x7f4e3800ba50 comp rx=0 tx=0).stop 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 shutdown_connections 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e400feb10 0x7f4e401948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e300a1260 0x7f4e300a3710 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e40100690 0x7f4e4019be50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 --2- 172.21.15.64:0/101551137 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e400ffd00 0x7f4e40194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.386 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 >> 172.21.15.64:0/101551137 conn(0x7f4e400fa130 msgr2=0x7f4e401031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 shutdown_connections 2023-07-14T18:24:31.387 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.303+0000 7f4e470c3700 1 -- 172.21.15.64:0/101551137 wait complete. 2023-07-14T18:24:31.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:24:31.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/382054649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 msgr2=0x7fc9ec0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/382054649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec0fef20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009990 tx=0x7fc9e8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/382054649 shutdown_connections 2023-07-14T18:24:31.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/382054649 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec100690 0x7fc9ec106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/382054649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec0fef20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/382054649 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec0ffd00 0x7fc9ec100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/382054649 >> 172.21.15.64:0/382054649 conn(0x7fc9ec0fa130 msgr2=0x7fc9ec0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/382054649 shutdown_connections 2023-07-14T18:24:31.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.463+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/382054649 wait complete. 2023-07-14T18:24:31.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 Processor -- start 2023-07-14T18:24:31.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 -- start start 2023-07-14T18:24:31.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec0ffd00 0x7fc9ec19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec100690 0x7fc9ec19cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19d3e0 con 0x7fc9ec0feb10 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19d550 con 0x7fc9ec100690 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f3e8b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec19d710 con 0x7fc9ec0ffd00 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48334/0 (socket says 172.21.15.64:48334) 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1904687784 learned_addr learned my addr 172.21.15.64:0/1904687784 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f2428700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec100690 0x7fc9ec19cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.464+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec0ffd00 0x7fc9ec19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1904687784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec0ffd00 msgr2=0x7fc9ec19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec0ffd00 0x7fc9ec19a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1904687784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec100690 msgr2=0x7fc9ec19cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec100690 0x7fc9ec19cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1904687784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e80095f0 con 0x7fc9ec0feb10 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800b5c0 tx=0x7fc9e8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9deffd700 1 -- 172.21.15.64:0/1904687784 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e801d070 con 0x7fc9ec0feb10 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec19d990 con 0x7fc9ec0feb10 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9deffd700 1 -- 172.21.15.64:0/1904687784 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9e8004080 con 0x7fc9ec0feb10 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9deffd700 1 -- 172.21.15.64:0/1904687784 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8022cd0 con 0x7fc9ec0feb10 2023-07-14T18:24:31.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.465+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec19de10 con 0x7fc9ec0feb10 2023-07-14T18:24:31.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.467+0000 7fc9deffd700 1 -- 172.21.15.64:0/1904687784 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc9e8022470 con 0x7fc9ec0feb10 2023-07-14T18:24:31.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.468+0000 7fc9deffd700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 0x7fc9d80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.468+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 0x7fc9d80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.468+0000 7fc9deffd700 1 -- 172.21.15.64:0/1904687784 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9e80ce3b0 con 0x7fc9ec0feb10 2023-07-14T18:24:31.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.468+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 0x7fc9d80a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0005950 tx=0x7fc9e00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.468+0000 7fca00023b80 1 Processor -- start 2023-07-14T18:24:31.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 -- start start 2023-07-14T18:24:31.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db04f10 0x564e1db05320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db05e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1da09c60 con 0x564e1db06460 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1da5c9f0 con 0x564e1db04f10 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fca00023b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1da55e80 con 0x564e1db05860 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48346/0 (socket says 172.21.15.64:48346) 2023-07-14T18:24:31.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 -- 172.21.15.64:0/1326204544 learned_addr learned my addr 172.21.15.64:0/1326204544 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db04f10 0x564e1db05320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db05e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 -- 172.21.15.64:0/1326204544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 msgr2=0x564e1db05e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db05e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 -- 172.21.15.64:0/1326204544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db04f10 msgr2=0x564e1db05320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db04f10 0x564e1db05320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 -- 172.21.15.64:0/1326204544 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e1d883d90 con 0x564e1db06460 2023-07-14T18:24:31.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.469+0000 7fc9f2428700 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009dd0 tx=0x7fc9e40077e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27415689c82ede0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fc9d7fff700 1 -- 172.21.15.64:0/1326204544 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4018070 con 0x564e1db06460 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fc9d7fff700 1 -- 172.21.15.64:0/1326204544 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9e400f050 con 0x564e1db06460 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fc9d7fff700 1 -- 172.21.15.64:0/1326204544 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e4004830 con 0x564e1db06460 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 -- 172.21.15.64:0/1326204544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 msgr2=0x564e1db068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc9e4009dd0 tx=0x7fc9e40077e0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 -- 172.21.15.64:0/1326204544 shutdown_connections 2023-07-14T18:24:31.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db06460 0x564e1db068d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db04f10 0x564e1db05320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 --2- 172.21.15.64:0/1326204544 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db05e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 -- 172.21.15.64:0/1326204544 >> 172.21.15.64:0/1326204544 conn(0x564e1dafa630 msgr2=0x564e1daff980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 -- 172.21.15.64:0/1326204544 shutdown_connections 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.470+0000 7fca00023b80 1 -- 172.21.15.64:0/1326204544 wait complete. 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 -- start start 2023-07-14T18:24:31.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 Processor -- start 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db0edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db15e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1d9f5a70 con 0x564e1db04f10 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1d9f5770 con 0x564e1db06460 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.471+0000 7fca00023b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1d8d98d0 con 0x564e1db05860 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fca00023b80 10 client.0 _ll_register_callbacks cb 0x564e1d9f3710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48348/0 (socket says 172.21.15.64:48348) 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 -- 172.21.15.64:0/401928645 learned_addr learned my addr 172.21.15.64:0/401928645 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f2428700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db15e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1426700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db0edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 -- 172.21.15.64:0/401928645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 msgr2=0x564e1db0edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db0edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.534 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 -- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 msgr2=0x564e1db15e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db15e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 -- 172.21.15.64:0/401928645 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e4009a00 con 0x564e1db04f10 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc9e809b200 tx=0x7fc9e8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b165c1e0d9a2f132 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a38c0 con 0x564e1db04f10 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9e80998a0 con 0x564e1db04f10 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e1d9af4a0 con 0x564e1db04f10 2023-07-14T18:24:31.535 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9e80cbca0 con 0x564e1db04f10 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fca00023b80 10 client.15303 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.472+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564e1d86cc10 con 0x564e1db04f10 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x564e1d9af4a0 con 0x564e1db04f10 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc9e809f070 con 0x564e1db04f10 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 10 client.15303 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 10 client.15303 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 20 client.15303 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.536 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db197b0 0x564e1db18d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 4 client.15303 mds_command: new command op to 24545 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e1dad3d50 con 0x564e1db197b0 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fc9f1426700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db197b0 0x564e1db18d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.473+0000 7fc9f1426700 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db197b0 0x564e1db18d60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8f955e78f1e106 server_cookie=20b6e2998ad2b1ba in_seq=0 out_seq=0 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fc9d57fa700 10 client.15303 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fc9d57fa700 1 -- 172.21.15.64:0/401928645 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x564e1dad3d50 con 0x564e1db197b0 2023-07-14T18:24:31.537 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fc9d57fa700 10 client.15303 handle_command_reply: tid=0 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 client.15303 shutdown 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 msgr2=0x564e1db0e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc9e809b200 tx=0x7fc9e8009310 comp rx=0 tx=0).stop 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 shutdown_connections 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db197b0 0x564e1db18d60 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db15e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db0e8b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.538 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 --2- 172.21.15.64:0/401928645 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db0edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fc9d57fa700 0 client.15303 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:31.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 >> 172.21.15.64:0/401928645 conn(0x564e1dafa630 msgr2=0x564e1dafe600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 shutdown_connections 2023-07-14T18:24:31.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.474+0000 7fca00023b80 1 -- 172.21.15.64:0/401928645 wait complete. 2023-07-14T18:24:31.539 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.475+0000 7fca00023b80 20 client.15303 trim_cache size 0 max 16384 2023-07-14T18:24:31.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.492+0000 7fca00023b80 1 Processor -- start 2023-07-14T18:24:31.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.492+0000 7fca00023b80 1 -- start start 2023-07-14T18:24:31.546 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.492+0000 7fca00023b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.492+0000 7fca00023b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 0x564e1db15df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fca00023b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db06460 0x564e1db16600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fca00023b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1db01510 con 0x564e1db04f10 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fca00023b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1db0f7b0 con 0x564e1db05860 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fca00023b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1dad3d50 con 0x564e1db06460 2023-07-14T18:24:31.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48356/0 (socket says 172.21.15.64:48356) 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1703872558 learned_addr learned my addr 172.21.15.64:0/1703872558 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 0x564e1db15df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f2428700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db06460 0x564e1db16600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1703872558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db06460 msgr2=0x564e1db16600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db06460 0x564e1db16600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1703872558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 msgr2=0x564e1db15df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 0x564e1db15df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 -- 172.21.15.64:0/1703872558 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e1d8aa200 con 0x564e1db04f10 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1426700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 0x564e1db15df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc9e809b490 tx=0x7fc9e8097970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcd23f482b2b3355 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9dcff9700 1 -- 172.21.15.64:0/1703872558 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a1180 con 0x564e1db04f10 2023-07-14T18:24:31.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9dcff9700 1 -- 172.21.15.64:0/1703872558 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9e80a38c0 con 0x564e1db04f10 2023-07-14T18:24:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.493+0000 7fc9dcff9700 1 -- 172.21.15.64:0/1703872558 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a3430 con 0x564e1db04f10 2023-07-14T18:24:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- 172.21.15.64:0/1703872558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 msgr2=0x564e1daff0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc9e809b490 tx=0x7fc9e8097970 comp rx=0 tx=0).stop 2023-07-14T18:24:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- 172.21.15.64:0/1703872558 shutdown_connections 2023-07-14T18:24:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db06460 0x564e1db16600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1daff0a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 --2- 172.21.15.64:0/1703872558 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db05860 0x564e1db15df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- 172.21.15.64:0/1703872558 >> 172.21.15.64:0/1703872558 conn(0x564e1dafa630 msgr2=0x564e1dafbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- 172.21.15.64:0/1703872558 shutdown_connections 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- 172.21.15.64:0/1703872558 wait complete. 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 -- start start 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.494+0000 7fca00023b80 1 Processor -- start 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db11500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1db14a10 con 0x564e1db04f10 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1db14b80 con 0x564e1db06460 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e1db14d40 con 0x564e1db05860 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fca00023b80 10 client.0 _ll_register_callbacks cb 0x7fc9e809ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48362/0 (socket says 172.21.15.64:48362) 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 -- 172.21.15.64:0/3507342261 learned_addr learned my addr 172.21.15.64:0/3507342261 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f2428700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1426700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db11500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 -- 172.21.15.64:0/3507342261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 msgr2=0x564e1db11500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db11500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 -- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 msgr2=0x564e1db144d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:31.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db144d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 -- 172.21.15.64:0/3507342261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e1d86cc10 con 0x564e1db04f10 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9f1c27700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8097f20 tx=0x7fc9e80cbe80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114bb3343c95fe12 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9d57fa700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e80a10b0 con 0x564e1db04f10 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.495+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc9e8067280 con 0x564e1db04f10 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9e400f050 con 0x564e1db04f10 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9e8067440 con 0x564e1db04f10 2023-07-14T18:24:31.557 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 10 client.15309 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564e1d882260 con 0x564e1db04f10 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc9e8083030 con 0x564e1db04f10 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc9e80972f0 con 0x564e1db04f10 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 10 client.15309 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 10 client.15309 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 20 client.15309 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:31.558 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db723e0 0x564e1db6f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 4 client.15309 mds_command: new command op to 24545 tid=0[{"prefix": "status"}] 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564e1da4ee60 con 0x564e1db723e0 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.496+0000 7fc9f1426700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db723e0 0x564e1db6f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fc9d57fa700 10 client.15309 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fc9f1426700 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db723e0 0x564e1db6f420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd0f8bc2bb06508d server_cookie=606f7e8cedfe2546 in_seq=0 out_seq=0 2023-07-14T18:24:31.559 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fc9d57fa700 1 -- 172.21.15.64:0/3507342261 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x564e1da4ee60 con 0x564e1db723e0 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fc9d57fa700 10 client.15309 handle_command_reply: tid=0 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 client.15309 shutdown 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 msgr2=0x564e1db10fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8097f20 tx=0x7fc9e80cbe80 comp rx=0 tx=0).stop 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 shutdown_connections 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x564e1db723e0 0x564e1db6f420 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x564e1db06460 0x564e1db144d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.560 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e1db04f10 0x564e1db10fc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fc9d57fa700 0 client.15309 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 --2- 172.21.15.64:0/3507342261 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x564e1db05860 0x564e1db11500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 >> 172.21.15.64:0/3507342261 conn(0x564e1dafa630 msgr2=0x564e1daff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 shutdown_connections 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 1 -- 172.21.15.64:0/3507342261 wait complete. 2023-07-14T18:24:31.561 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.497+0000 7fca00023b80 20 client.15309 trim_cache size 0 max 16384 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24545, 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:31.562 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 86.082246663000006, 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 92.187349916000002 2023-07-14T18:24:31.563 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:31.564 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 msgr2=0x7fc9d80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 0x7fc9d80a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0005950 tx=0x7fc9e00058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 msgr2=0x7fc9ec068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc9e800b5c0 tx=0x7fc9e8005ef0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 shutdown_connections 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc9d80a11e0 0x7fc9d80a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc9ec100690 0x7fc9ec19cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.565 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9ec0feb10 0x7fc9ec068310 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 --2- 172.21.15.64:0/1904687784 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc9ec0ffd00 0x7fc9ec19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:31.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 >> 172.21.15.64:0/1904687784 conn(0x7fc9ec0fa130 msgr2=0x7fc9ec0fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:31.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 shutdown_connections 2023-07-14T18:24:31.566 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:31.498+0000 7fc9f3e8b700 1 -- 172.21.15.64:0/1904687784 wait complete. 2023-07-14T18:24:31.567 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T18:24:31.568 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T18:24:31.580 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:24:31.581 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T18:24:31.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:31 smithi064 ceph-mon[96720]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2023-07-14T18:24:31.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:31 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1243986030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:31.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:31 smithi144 ceph-mon[98417]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2023-07-14T18:24:31.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:31 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1243986030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:31.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:31 smithi105 ceph-mon[98013]: pgmap v299: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2023-07-14T18:24:31.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:31 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1243986030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:32.140 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:33.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 -- 172.21.15.64:0/373122960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 msgr2=0x7f90ec0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 --2- 172.21.15.64:0/373122960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec0fefc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009990 tx=0x7f90dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 -- 172.21.15.64:0/373122960 shutdown_connections 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 --2- 172.21.15.64:0/373122960 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec100740 0x7f90ec107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 --2- 172.21.15.64:0/373122960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec0fefc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 --2- 172.21.15.64:0/373122960 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 -- 172.21.15.64:0/373122960 >> 172.21.15.64:0/373122960 conn(0x7f90ec0fa1e0 msgr2=0x7f90ec0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:33.352 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 -- 172.21.15.64:0/373122960 shutdown_connections 2023-07-14T18:24:33.353 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.298+0000 7f90f1b25700 1 -- 172.21.15.64:0/373122960 wait complete. 2023-07-14T18:24:33.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 Processor -- start 2023-07-14T18:24:33.354 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 -- start start 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec100740 0x7f90ec19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec19d450 con 0x7f90ec0febb0 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec19d5c0 con 0x7f90ec0ffdb0 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90f1b25700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec19d780 con 0x7f90ec100740 2023-07-14T18:24:33.355 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90eb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.299+0000 7f90eb7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48376/0 (socket says 172.21.15.64:48376) 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 -- 172.21.15.64:0/1403507639 learned_addr learned my addr 172.21.15.64:0/1403507639 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90ebfff700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec100740 0x7f90ec19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eaffd700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 -- 172.21.15.64:0/1403507639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec100740 msgr2=0x7f90ec19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec100740 0x7f90ec19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.356 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 -- 172.21.15.64:0/1403507639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 msgr2=0x7f90ec19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 -- 172.21.15.64:0/1403507639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc0095f0 con 0x7f90ec0febb0 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eaffd700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90eb7fe700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f90dc004730 tx=0x7f90dc004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc01d070 con 0x7f90ec0febb0 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f90dc004cc0 con 0x7f90ec0febb0 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec19da00 con 0x7f90ec0febb0 2023-07-14T18:24:33.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90dc005280 con 0x7f90ec0febb0 2023-07-14T18:24:33.358 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.300+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec19ddb0 con 0x7f90ec0febb0 2023-07-14T18:24:33.358 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.301+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec106240 con 0x7f90ec0febb0 2023-07-14T18:24:33.358 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.303+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f90dc003b70 con 0x7f90ec0febb0 2023-07-14T18:24:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.304+0000 7f90e8ff9700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 0x7f90d40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.304+0000 7f90eaffd700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 0x7f90d40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.304+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f90dc0ce110 con 0x7f90ec0febb0 2023-07-14T18:24:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.304+0000 7f90eaffd700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 0x7f90d40a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005fd0 tx=0x7f90e0005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:33.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.307+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f90dc097480 con 0x7f90ec0febb0 2023-07-14T18:24:33.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.520+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f90ec060a80 con 0x7f90d40a11e0 2023-07-14T18:24:33.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.521+0000 7f90e8ff9700 1 -- 172.21.15.64:0/1403507639 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f90ec060a80 con 0x7f90d40a11e0 2023-07-14T18:24:33.573 INFO:teuthology.orchestra.run.smithi064.stdout:Backend: cephadm 2023-07-14T18:24:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:Available: Yes 2023-07-14T18:24:33.574 INFO:teuthology.orchestra.run.smithi064.stdout:Paused: No 2023-07-14T18:24:33.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 msgr2=0x7f90d40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 0x7f90d40a3690 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f90e0005fd0 tx=0x7f90e0005ee0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 msgr2=0x7f90ec068c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f90dc004730 tx=0x7f90dc004760 comp rx=0 tx=0).stop 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 shutdown_connections 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90ec100740 0x7f90ec19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f90d40a11e0 0x7f90d40a3690 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90ec0febb0 0x7f90ec068c60 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 --2- 172.21.15.64:0/1403507639 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0ffdb0 0x7f90ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:33.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 >> 172.21.15.64:0/1403507639 conn(0x7f90ec0fa1e0 msgr2=0x7f90ec103380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:33.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 shutdown_connections 2023-07-14T18:24:33.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:33.523+0000 7f90f1b25700 1 -- 172.21.15.64:0/1403507639 wait complete. 2023-07-14T18:24:33.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:33 smithi064 ceph-mon[96720]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:33.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:33 smithi144 ceph-mon[98417]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:33.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:33 smithi105 ceph-mon[98013]: pgmap v300: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2023-07-14T18:24:34.008 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T18:24:34.611 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:35.401 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:35 smithi064 ceph-mon[96720]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:35.402 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:35 smithi064 ceph-mon[96720]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:35.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:35 smithi144 ceph-mon[98417]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:35.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:35 smithi144 ceph-mon[98417]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:35.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:35 smithi105 ceph-mon[98013]: pgmap v301: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2023-07-14T18:24:35.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:35 smithi105 ceph-mon[98013]: from='client.15315 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:35.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- 172.21.15.64:0/1837808138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 msgr2=0x7f2d98107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:35.813 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1837808138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d98107000 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f2d9000b0a0 tx=0x7f2d9000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- 172.21.15.64:0/1837808138 shutdown_connections 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1837808138 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d980fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1837808138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d98107000 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1837808138 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d98100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- 172.21.15.64:0/1837808138 >> 172.21.15.64:0/1837808138 conn(0x7f2d980fa1e0 msgr2=0x7f2d980fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:35.814 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- 172.21.15.64:0/1837808138 shutdown_connections 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- 172.21.15.64:0/1837808138 wait complete. 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 Processor -- start 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.760+0000 7f2d9f309700 1 -- start start 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d98068c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d9819a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:35.815 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9819d4d0 con 0x7f2d98100740 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9819d640 con 0x7f2d980ffdb0 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9f309700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9819d800 con 0x7f2d980febb0 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d0a5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d98068c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9c8a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d9819a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d0a5700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d98068c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:56044/0 (socket says 172.21.15.64:56044) 2023-07-14T18:24:35.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9c8a4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d9819a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.64:41928/0 (socket says 172.21.15.64:41928) 2023-07-14T18:24:35.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d0a5700 1 -- 172.21.15.64:0/1481958852 learned_addr learned my addr 172.21.15.64:0/1481958852 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:35.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48392/0 (socket says 172.21.15.64:48392) 2023-07-14T18:24:35.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 -- 172.21.15.64:0/1481958852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 msgr2=0x7f2d98068c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:35.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d98068c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 -- 172.21.15.64:0/1481958852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 msgr2=0x7f2d9819a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:35.818 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d9819a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 -- 172.21.15.64:0/1481958852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d90009d00 con 0x7f2d98100740 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.761+0000 7f2d9d8a6700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009fd0 tx=0x7f2d90004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.762+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000e040 con 0x7f2d98100740 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.762+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d9819da80 con 0x7f2d98100740 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.762+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2d900056e0 con 0x7f2d98100740 2023-07-14T18:24:35.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.762+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d9001dac0 con 0x7f2d98100740 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.762+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d9819def0 con 0x7f2d98100740 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.763+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d9810a280 con 0x7f2d98100740 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.764+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2d90005850 con 0x7f2d98100740 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.764+0000 7f2d8e7fc700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 0x7f2d840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.765+0000 7f2d9d0a5700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 0x7f2d840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.765+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2d900ce2c0 con 0x7f2d98100740 2023-07-14T18:24:35.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.765+0000 7f2d9d0a5700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 0x7f2d840a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400bd70 tx=0x7f2d9400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:35.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.769+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2d90097630 con 0x7f2d98100740 2023-07-14T18:24:36.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.976+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f2d98060a80 con 0x7f2d840a11e0 2023-07-14T18:24:36.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.996+0000 7f2d8e7fc700 1 -- 172.21.15.64:0/1481958852 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f2d98060a80 con 0x7f2d840a11e0 2023-07-14T18:24:36.050 INFO:teuthology.orchestra.run.smithi064.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T18:24:36.050 INFO:teuthology.orchestra.run.smithi064.stdout:mds.a smithi064 running (99s) - 98s 17.5M - 18.0.0-4940-g81c9387d 219b180907a8 a11e7f51b664 2023-07-14T18:24:36.051 INFO:teuthology.orchestra.run.smithi064.stdout:mds.b smithi105 running (97s) - 96s 33.0M - 18.0.0-4940-g81c9387d 219b180907a8 7f3b040aee59 2023-07-14T18:24:36.051 INFO:teuthology.orchestra.run.smithi064.stdout:mds.c smithi144 running (108s) - 107s 21.2M - 18.0.0-4940-g81c9387d 219b180907a8 b965cece7763 2023-07-14T18:24:36.051 INFO:teuthology.orchestra.run.smithi064.stdout:mds.d smithi064 running (115s) - 114s 20.0M - 18.0.0-4940-g81c9387d 219b180907a8 fc7cc862323b 2023-07-14T18:24:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:mds.e smithi105 running (112s) - 112s 21.2M - 18.0.0-4940-g81c9387d 219b180907a8 0afefd87911e 2023-07-14T18:24:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:mds.f smithi144 running (110s) - 110s 18.4M - 18.0.0-4940-g81c9387d 219b180907a8 3779737864b3 2023-07-14T18:24:36.052 INFO:teuthology.orchestra.run.smithi064.stdout:mds.g smithi064 running (101s) - 101s 19.9M - 18.0.0-4940-g81c9387d 219b180907a8 c8fb439b7b05 2023-07-14T18:24:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:mds.h smithi105 running (106s) - 105s 18.1M - 18.0.0-4940-g81c9387d 219b180907a8 093ddda7cff0 2023-07-14T18:24:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:mds.i smithi144 running (103s) - 103s 18.3M - 18.0.0-4940-g81c9387d 219b180907a8 20d27cab515d 2023-07-14T18:24:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:mds.j smithi064 running (117s) - 116s 19.7M - 18.0.0-4940-g81c9387d 219b180907a8 ab5a6a847d36 2023-07-14T18:24:36.053 INFO:teuthology.orchestra.run.smithi064.stdout:mds.k smithi105 running (119s) - 119s 14.1M - 18.0.0-4940-g81c9387d 219b180907a8 6a43c89448fc 2023-07-14T18:24:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:mds.l smithi144 running (2m) - 2m 17.5M - 18.0.0-4940-g81c9387d 219b180907a8 d8697520074b 2023-07-14T18:24:36.054 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.x smithi064 *:9283,8765 running (8m) - 8m 518M - 18.0.0-4940-g81c9387d 219b180907a8 c9d8d66dde85 2023-07-14T18:24:36.055 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.y smithi105 *:8443,8765 running (6m) - 6m 446M - 18.0.0-4940-g81c9387d 219b180907a8 9bc5d10f7588 2023-07-14T18:24:36.055 INFO:teuthology.orchestra.run.smithi064.stdout:mgr.z smithi144 *:8443,8765 running (6m) - 6m 446M - 18.0.0-4940-g81c9387d 219b180907a8 a6a0f3806edb 2023-07-14T18:24:36.055 INFO:teuthology.orchestra.run.smithi064.stdout:mon.a smithi064 running (8m) - 8m 97.2M 2048M 18.0.0-4940-g81c9387d 219b180907a8 e874a9b4921e 2023-07-14T18:24:36.055 INFO:teuthology.orchestra.run.smithi064.stdout:mon.b smithi105 running (6m) - 6m 61.9M 2048M 18.0.0-4940-g81c9387d 219b180907a8 4f07c1d47ce2 2023-07-14T18:24:36.056 INFO:teuthology.orchestra.run.smithi064.stdout:mon.c smithi144 running (6m) - 6m 65.2M 2048M 18.0.0-4940-g81c9387d 219b180907a8 dc66a4d8a24e 2023-07-14T18:24:36.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 smithi064 running (6m) - 6m 69.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 a1e602361bcf 2023-07-14T18:24:36.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 smithi105 running (5m) - 5m 75.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 694907ead8d9 2023-07-14T18:24:36.056 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 smithi144 running (5m) - 5m 74.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 8e361e1b9644 2023-07-14T18:24:36.057 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 smithi064 running (5m) - 5m 65.0M 4292M 18.0.0-4940-g81c9387d 219b180907a8 816e02a4c894 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 smithi105 running (4m) - 4m 62.0M 4292M 18.0.0-4940-g81c9387d 219b180907a8 69aa2632ba16 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 smithi144 running (4m) - 4m 67.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 9ce93e902371 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 smithi064 running (4m) - 4m 60.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 cc0dd66d4746 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 smithi105 running (3m) - 3m 64.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 c7c9ef1dd3e1 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 smithi144 running (3m) - 3m 61.8M 4292M 18.0.0-4940-g81c9387d 219b180907a8 370ea55cd230 2023-07-14T18:24:36.058 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 smithi064 running (3m) - 3m 63.4M 4292M 18.0.0-4940-g81c9387d 219b180907a8 00fb15992b6e 2023-07-14T18:24:36.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 smithi105 running (2m) - 2m 62.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 399a4b14cc51 2023-07-14T18:24:36.059 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 smithi144 running (2m) - 2m 53.6M 4292M 18.0.0-4940-g81c9387d 219b180907a8 8d466112fd98 2023-07-14T18:24:36.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 msgr2=0x7f2d840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 0x7f2d840a3690 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400bd70 tx=0x7f2d9400b4a0 comp rx=0 tx=0).stop 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 msgr2=0x7f2d9819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f2d90009fd0 tx=0x7f2d90004c80 comp rx=0 tx=0).stop 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 shutdown_connections 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2d980febb0 0x7f2d98068c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:36.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f2d840a11e0 0x7f2d840a3690 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:36.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d98100740 0x7f2d9819cd30 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:36.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 --2- 172.21.15.64:0/1481958852 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2d980ffdb0 0x7f2d9819a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:36.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 >> 172.21.15.64:0/1481958852 conn(0x7f2d980fa1e0 msgr2=0x7f2d980fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:36.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 shutdown_connections 2023-07-14T18:24:36.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:35.999+0000 7f2d9f309700 1 -- 172.21.15.64:0/1481958852 wait complete. 2023-07-14T18:24:36.480 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T18:24:37.040 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:37.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:37 smithi064 ceph-mon[96720]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:37.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:37 smithi064 ceph-mon[96720]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:37.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:37 smithi064 ceph-mon[96720]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:24:37.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:37 smithi144 ceph-mon[98417]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:37.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:37 smithi144 ceph-mon[98417]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:37.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:37 smithi144 ceph-mon[98417]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:24:37.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:37 smithi105 ceph-mon[98013]: pgmap v302: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2023-07-14T18:24:37.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:37 smithi105 ceph-mon[98013]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:37.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:37 smithi105 ceph-mon[98013]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T18:24:38.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 -- 172.21.15.64:0/4214154667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 msgr2=0x7ff7c40fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:38.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/4214154667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c40fefc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4009990 tx=0x7ff7b4009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 -- 172.21.15.64:0/4214154667 shutdown_connections 2023-07-14T18:24:38.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/4214154667 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c4100740 0x7ff7c4107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/4214154667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c40fefc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/4214154667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c40ffdb0 0x7ff7c4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 -- 172.21.15.64:0/4214154667 >> 172.21.15.64:0/4214154667 conn(0x7ff7c40fa1e0 msgr2=0x7ff7c40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:38.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 -- 172.21.15.64:0/4214154667 shutdown_connections 2023-07-14T18:24:38.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.196+0000 7ff7c9ada700 1 -- 172.21.15.64:0/4214154667 wait complete. 2023-07-14T18:24:38.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.197+0000 7ff7c9ada700 1 Processor -- start 2023-07-14T18:24:38.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.197+0000 7ff7c9ada700 1 -- start start 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40ffdb0 0x7ff7c419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c419d420 con 0x7ff7c40febb0 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c419d590 con 0x7ff7c4100740 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c9ada700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c419d750 con 0x7ff7c40ffdb0 2023-07-14T18:24:38.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c3fff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.64:41944/0 (socket says 172.21.15.64:41944) 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48402/0 (socket says 172.21.15.64:48402) 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c3fff700 1 -- 172.21.15.64:0/1228324301 learned_addr learned my addr 172.21.15.64:0/1228324301 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c2ffd700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40ffdb0 0x7ff7c419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 -- 172.21.15.64:0/1228324301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40ffdb0 msgr2=0x7ff7c419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40ffdb0 0x7ff7c419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 -- 172.21.15.64:0/1228324301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 msgr2=0x7ff7c419cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 -- 172.21.15.64:0/1228324301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b40095f0 con 0x7ff7c40febb0 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c3fff700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.198+0000 7ff7c37fe700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400c010 tx=0x7ff7b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.199+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b401d070 con 0x7ff7c40febb0 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.199+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff7b4004c70 con 0x7ff7c40febb0 2023-07-14T18:24:38.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.199+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c419d9d0 con 0x7ff7c40febb0 2023-07-14T18:24:38.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.199+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b4003a30 con 0x7ff7c40febb0 2023-07-14T18:24:38.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.199+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c419dd80 con 0x7ff7c40febb0 2023-07-14T18:24:38.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.200+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c410a220 con 0x7ff7c40febb0 2023-07-14T18:24:38.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.201+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff7b4005020 con 0x7ff7c40febb0 2023-07-14T18:24:38.257 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.201+0000 7ff7c0ff9700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 0x7ff7ac0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:38.257 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.201+0000 7ff7c2ffd700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 0x7ff7ac0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:38.257 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.201+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff7b40cd870 con 0x7ff7c40febb0 2023-07-14T18:24:38.257 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.202+0000 7ff7c2ffd700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 0x7ff7ac0a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40691e0 tx=0x7ff7b800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:38.258 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.206+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7b4096d40 con 0x7ff7c40febb0 2023-07-14T18:24:38.467 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.414+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7c4060a80 con 0x7ff7ac0a1150 2023-07-14T18:24:38.473 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.421+0000 7ff7c0ff9700 1 -- 172.21.15.64:0/1228324301 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7ff7c4060a80 con 0x7ff7ac0a1150 2023-07-14T18:24:38.474 INFO:teuthology.orchestra.run.smithi064.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T18:24:38.474 INFO:teuthology.orchestra.run.smithi064.stdout:mds.all 12/12 - 2m smithi064=a;smithi064=d;smithi064=g;smithi064=j;smithi105=b;smithi105=e;smithi105=h;smithi105=k;smithi144=c;smithi144=f;smithi144=i;smithi144=l;count:12 2023-07-14T18:24:38.475 INFO:teuthology.orchestra.run.smithi064.stdout:mgr 3/3 - 6m smithi064=x;smithi105=y;smithi144=z;count:3 2023-07-14T18:24:38.475 INFO:teuthology.orchestra.run.smithi064.stdout:mon 3/3 - 6m smithi064:172.21.15.64=a;smithi105:172.21.15.105=b;smithi144:172.21.15.144=c;count:3 2023-07-14T18:24:38.475 INFO:teuthology.orchestra.run.smithi064.stdout:osd 12 - - 2023-07-14T18:24:38.476 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 msgr2=0x7ff7ac0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:38.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 0x7ff7ac0a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40691e0 tx=0x7ff7b800b410 comp rx=0 tx=0).stop 2023-07-14T18:24:38.477 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 msgr2=0x7ff7c4068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:38.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff7b400c010 tx=0x7ff7b4004730 comp rx=0 tx=0).stop 2023-07-14T18:24:38.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 shutdown_connections 2023-07-14T18:24:38.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff7ac0a1150 0x7ff7ac0a3600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.478 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff7c4100740 0x7ff7c419cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7c40febb0 0x7ff7c4068c80 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 --2- 172.21.15.64:0/1228324301 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff7c40ffdb0 0x7ff7c419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:38.479 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 >> 172.21.15.64:0/1228324301 conn(0x7ff7c40fa1e0 msgr2=0x7ff7c40fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:38.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 shutdown_connections 2023-07-14T18:24:38.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:38.424+0000 7ff7c9ada700 1 -- 172.21.15.64:0/1228324301 wait complete. 2023-07-14T18:24:38.894 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T18:24:39.453 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:39.650 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:39 smithi064 ceph-mon[96720]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:24:39.650 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:39 smithi064 ceph-mon[96720]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:39.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:39 smithi144 ceph-mon[98417]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:24:39.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:39 smithi144 ceph-mon[98417]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:39.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:39 smithi105 ceph-mon[98013]: pgmap v303: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 10 op/s 2023-07-14T18:24:39.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:39 smithi105 ceph-mon[98013]: from='client.15327 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 -- 172.21.15.64:0/4243399863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd8100740 msgr2=0x7f7cd8107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/4243399863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd8100740 0x7f7cd8107000 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000b0a0 tx=0x7f7cd000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 -- 172.21.15.64:0/4243399863 shutdown_connections 2023-07-14T18:24:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/4243399863 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd80febb0 0x7f7cd80fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/4243399863 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd8100740 0x7f7cd8107000 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/4243399863 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd8100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 -- 172.21.15.64:0/4243399863 >> 172.21.15.64:0/4243399863 conn(0x7f7cd80fa1e0 msgr2=0x7f7cd80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 -- 172.21.15.64:0/4243399863 shutdown_connections 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.599+0000 7f7cdedd4700 1 -- 172.21.15.64:0/4243399863 wait complete. 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 Processor -- start 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 -- start start 2023-07-14T18:24:40.654 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd8100740 0x7f7cd819ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd819d470 con 0x7f7cd80febb0 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd819d5e0 con 0x7f7cd80ffdb0 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdedd4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd819d7a0 con 0x7f7cd8100740 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdcb70700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdcb70700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48416/0 (socket says 172.21.15.64:48416) 2023-07-14T18:24:40.655 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.600+0000 7f7cdcb70700 1 -- 172.21.15.64:0/3472934159 learned_addr learned my addr 172.21.15.64:0/3472934159 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdd371700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd8100740 0x7f7cd819ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cd7fff700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 -- 172.21.15.64:0/3472934159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd8100740 msgr2=0x7f7cd819ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd8100740 0x7f7cd819ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 -- 172.21.15.64:0/3472934159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 msgr2=0x7f7cd819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 -- 172.21.15.64:0/3472934159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd0009d00 con 0x7f7cd80febb0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cd7fff700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdcb70700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c970 tx=0x7f7ccc00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc004220 con 0x7f7cd80febb0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ccc007d30 con 0x7f7cd80febb0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd819da80 con 0x7f7cd80febb0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc003bf0 con 0x7f7cd80febb0 2023-07-14T18:24:40.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.601+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd819df80 con 0x7f7cd80febb0 2023-07-14T18:24:40.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.602+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd8106130 con 0x7f7cd80febb0 2023-07-14T18:24:40.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.603+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7ccc0078a0 con 0x7f7cd80febb0 2023-07-14T18:24:40.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.604+0000 7f7cd5ffb700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 0x7f7cc80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:40.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.604+0000 7f7cd7fff700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 0x7f7cc80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:40.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.604+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7ccc0cc8b0 con 0x7f7cd80febb0 2023-07-14T18:24:40.659 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.605+0000 7f7cd7fff700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 0x7f7cc80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005fd0 tx=0x7f7cc4005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:40.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.608+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ccc095cf0 con 0x7f7cd80febb0 2023-07-14T18:24:40.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.809+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7cd8060a80 con 0x7f7cc80a1150 2023-07-14T18:24:40.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.811+0000 7f7cd5ffb700 1 -- 172.21.15.64:0/3472934159 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f7cd8060a80 con 0x7f7cc80a1150 2023-07-14T18:24:40.863 INFO:teuthology.orchestra.run.smithi064.stdout:HOST ADDR LABELS STATUS 2023-07-14T18:24:40.863 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 172.21.15.64 2023-07-14T18:24:40.863 INFO:teuthology.orchestra.run.smithi064.stdout:smithi105 172.21.15.105 2023-07-14T18:24:40.863 INFO:teuthology.orchestra.run.smithi064.stdout:smithi144 172.21.15.144 2023-07-14T18:24:40.864 INFO:teuthology.orchestra.run.smithi064.stdout:3 hosts in cluster 2023-07-14T18:24:40.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 msgr2=0x7f7cc80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:40.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 0x7f7cc80a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4005fd0 tx=0x7f7cc4005e20 comp rx=0 tx=0).stop 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 msgr2=0x7f7cd8068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc00c970 tx=0x7f7ccc00cd30 comp rx=0 tx=0).stop 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 shutdown_connections 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7cd8100740 0x7f7cd819ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7cc80a1150 0x7f7cc80a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7cd80febb0 0x7f7cd8068c80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 --2- 172.21.15.64:0/3472934159 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7cd80ffdb0 0x7f7cd819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:40.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.813+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 >> 172.21.15.64:0/3472934159 conn(0x7f7cd80fa1e0 msgr2=0x7f7cd8103270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:40.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.814+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 shutdown_connections 2023-07-14T18:24:40.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:40.814+0000 7f7cdedd4700 1 -- 172.21.15.64:0/3472934159 wait complete. 2023-07-14T18:24:41.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T18:24:41.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:41 smithi064 ceph-mon[96720]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T18:24:41.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:41 smithi144 ceph-mon[98417]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T18:24:41.774 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:41.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:41 smithi105 ceph-mon[98013]: pgmap v304: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 11 op/s 2023-07-14T18:24:42.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:42 smithi064 ceph-mon[96720]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:42.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:42 smithi144 ceph-mon[98417]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:42.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:42 smithi105 ceph-mon[98013]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:42.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 -- 172.21.15.64:0/191630305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0febb0 msgr2=0x7f436c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:42.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 --2- 172.21.15.64:0/191630305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0febb0 0x7f436c0fefc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f435c009990 tx=0x7f435c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 -- 172.21.15.64:0/191630305 shutdown_connections 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 --2- 172.21.15.64:0/191630305 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 --2- 172.21.15.64:0/191630305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0febb0 0x7f436c0fefc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 --2- 172.21.15.64:0/191630305 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0ffdb0 0x7f436c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 -- 172.21.15.64:0/191630305 >> 172.21.15.64:0/191630305 conn(0x7f436c0fa1e0 msgr2=0x7f436c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:42.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 -- 172.21.15.64:0/191630305 shutdown_connections 2023-07-14T18:24:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 -- 172.21.15.64:0/191630305 wait complete. 2023-07-14T18:24:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.916+0000 7f4372224700 1 Processor -- start 2023-07-14T18:24:42.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 -- start start 2023-07-14T18:24:42.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0febb0 0x7f436c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:42.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:42.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c19f6c0 con 0x7f436c0ffdb0 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c19f800 con 0x7f436c0febb0 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f4372224700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c19f990 con 0x7f436c100740 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48430/0 (socket says 172.21.15.64:48430) 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 -- 172.21.15.64:0/1184143081 learned_addr learned my addr 172.21.15.64:0/1184143081 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:42.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436bfff700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436b7fe700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0febb0 0x7f436c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 -- 172.21.15.64:0/1184143081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 msgr2=0x7f436c19f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c19f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 -- 172.21.15.64:0/1184143081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0febb0 msgr2=0x7f436c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0febb0 0x7f436c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436affd700 1 -- 172.21.15.64:0/1184143081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435c0095f0 con 0x7f436c0ffdb0 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.917+0000 7f436bfff700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:42.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f436affd700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f436000d980 tx=0x7f436000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4360009b40 con 0x7f436c0ffdb0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c19fc70 con 0x7f436c0ffdb0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f436000b6e0 con 0x7f436c0ffdb0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436000f620 con 0x7f436c0ffdb0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.918+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c1a01c0 con 0x7f436c0ffdb0 2023-07-14T18:24:42.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.919+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c10a2a0 con 0x7f436c0ffdb0 2023-07-14T18:24:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.920+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4360010460 con 0x7f436c0ffdb0 2023-07-14T18:24:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.921+0000 7f4368ff9700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 0x7f43540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.921+0000 7f436b7fe700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 0x7f43540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:42.975 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.921+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f43600ccb10 con 0x7f436c0ffdb0 2023-07-14T18:24:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.921+0000 7f436b7fe700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 0x7f43540a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f435c00b5c0 tx=0x7f435c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:42.976 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:42.925+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4360095ef0 con 0x7f436c0ffdb0 2023-07-14T18:24:43.180 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.128+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f436c068830 con 0x7f43540a1150 2023-07-14T18:24:43.182 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.131+0000 7f4368ff9700 1 -- 172.21.15.64:0/1184143081 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f436c068830 con 0x7f43540a1150 2023-07-14T18:24:43.183 INFO:teuthology.orchestra.run.smithi064.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T18:24:43.183 INFO:teuthology.orchestra.run.smithi064.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 3m ago LVM detected, locked 2023-07-14T18:24:43.184 INFO:teuthology.orchestra.run.smithi064.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 2m ago LVM detected, locked 2023-07-14T18:24:43.184 INFO:teuthology.orchestra.run.smithi064.stdout:smithi144 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050085400BGN 372G No 2m ago LVM detected, locked 2023-07-14T18:24:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 msgr2=0x7f43540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:43.185 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 0x7f43540a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f435c00b5c0 tx=0x7f435c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 msgr2=0x7f436c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f436000d980 tx=0x7f436000dd40 comp rx=0 tx=0).stop 2023-07-14T18:24:43.186 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 shutdown_connections 2023-07-14T18:24:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f436c100740 0x7f436c19f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f43540a1150 0x7f43540a3600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f436c0febb0 0x7f436c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:43.187 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 --2- 172.21.15.64:0/1184143081 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f436c0ffdb0 0x7f436c198070 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 >> 172.21.15.64:0/1184143081 conn(0x7f436c0fa1e0 msgr2=0x7f436c068120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 shutdown_connections 2023-07-14T18:24:43.188 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:43.133+0000 7f4372224700 1 -- 172.21.15.64:0/1184143081 wait complete. 2023-07-14T18:24:43.545 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:43 smithi064 ceph-mon[96720]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:24:43.627 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T18:24:43.639 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:24:43.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T18:24:43.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:43 smithi144 ceph-mon[98417]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:24:43.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:43 smithi105 ceph-mon[98013]: pgmap v305: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:24:44.196 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:44.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:44 smithi144 ceph-mon[98417]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:44.742 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:44 smithi064 ceph-mon[96720]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:44.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:44 smithi105 ceph-mon[98013]: from='client.15339 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:24:45.443 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 -- 172.21.15.64:0/2458256445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 msgr2=0x7f0944100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:45.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 --2- 172.21.15.64:0/2458256445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f0944100200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f0934009990 tx=0x7f0934009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 -- 172.21.15.64:0/2458256445 shutdown_connections 2023-07-14T18:24:45.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 --2- 172.21.15.64:0/2458256445 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f0944100740 0x7f0944107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 --2- 172.21.15.64:0/2458256445 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f09440febb0 0x7f09440fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 --2- 172.21.15.64:0/2458256445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f0944100200 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 -- 172.21.15.64:0/2458256445 >> 172.21.15.64:0/2458256445 conn(0x7f09440fa1e0 msgr2=0x7f09440fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 -- 172.21.15.64:0/2458256445 shutdown_connections 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.390+0000 7f094c143700 1 -- 172.21.15.64:0/2458256445 wait complete. 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 Processor -- start 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 -- start start 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09440febb0 0x7f0944068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:45.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0944100740 0x7f094419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.391+0000 7f094c143700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419d3d0 con 0x7f09440ffdb0 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f094c143700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419d540 con 0x7f0944100740 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f094c143700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f094419d700 con 0x7f09440febb0 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:45.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48442/0 (socket says 172.21.15.64:48442) 2023-07-14T18:24:45.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 -- 172.21.15.64:0/2399596949 learned_addr learned my addr 172.21.15.64:0/2399596949 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:45.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f0949edf700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09440febb0 0x7f0944068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:45.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f094a6e0700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0944100740 0x7f094419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:45.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 -- 172.21.15.64:0/2399596949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09440febb0 msgr2=0x7f0944068c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:45.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09440febb0 0x7f0944068c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 -- 172.21.15.64:0/2399596949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0944100740 msgr2=0x7f094419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:45.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0944100740 0x7f094419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 -- 172.21.15.64:0/2399596949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340095f0 con 0x7f09440ffdb0 2023-07-14T18:24:45.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f09496de700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0934006010 tx=0x7f0934004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:45.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093401d070 con 0x7f09440ffdb0 2023-07-14T18:24:45.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0934004000 con 0x7f09440ffdb0 2023-07-14T18:24:45.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0934022b90 con 0x7f09440ffdb0 2023-07-14T18:24:45.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f094419d980 con 0x7f09440ffdb0 2023-07-14T18:24:45.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.392+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f094419dd40 con 0x7f09440ffdb0 2023-07-14T18:24:45.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.393+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094410a220 con 0x7f09440ffdb0 2023-07-14T18:24:45.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.394+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0934003b70 con 0x7f09440ffdb0 2023-07-14T18:24:45.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.395+0000 7f093affd700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 0x7f09300a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:45.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.395+0000 7f0949edf700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 0x7f09300a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:45.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.395+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f09340cdde0 con 0x7f09440ffdb0 2023-07-14T18:24:45.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.395+0000 7f0949edf700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 0x7f09300a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f09440ffc10 tx=0x7f094000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:45.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.399+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f093405c780 con 0x7f09440ffdb0 2023-07-14T18:24:45.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:45 smithi064 ceph-mon[96720]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T18:24:45.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:45 smithi144 ceph-mon[98417]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T18:24:45.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.645+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f0944105b30 con 0x7f09440ffdb0 2023-07-14T18:24:45.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.646+0000 7f093affd700 1 -- 172.21.15.64:0/2399596949 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+2936 (secure 0 0 0) 0x7f093409ad40 con 0x7f09440ffdb0 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout:e27 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout:legacy client fscid: 1 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:45.698 INFO:teuthology.orchestra.run.smithi064.stdout:Filesystem 'cephfs' (1) 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:fs_name cephfs 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 27 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:created 2023-07-14T18:23:04.865157+0000 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2023-07-14T18:23:28.278497+0000 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:tableserver 0 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:root 0 2023-07-14T18:24:45.699 INFO:teuthology.orchestra.run.smithi064.stdout:session_timeout 300 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:session_autoclose 300 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:max_file_size 1099511627776 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:max_xattr_size 65536 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:required_client_features {} 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure 0 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:last_failure_osd_epoch 0 2023-07-14T18:24:45.700 INFO:teuthology.orchestra.run.smithi064.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:max_mds 5 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:in 0,1,2,3,4 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:up {0=24545,1=24508,2=24500,3=24487,4=24478} 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:failed 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:damaged 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:stopped 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:data_pools [3,4] 2023-07-14T18:24:45.701 INFO:teuthology.orchestra.run.smithi064.stdout:metadata_pool 2 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:inline_data disabled 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:balancer 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:bal_rank_mask -1 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:standby_count_wanted 0 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.b{0:24545} state up:active seq 3 addr [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.e{0:24464} state up:standby-replay seq 1 addr [v2:172.21.15.105:6834/2365162845,v1:172.21.15.105:6835/2365162845] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.i{1:24508} state up:active seq 9 addr [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.702 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.a{1:14628} state up:standby-replay seq 1 addr [v2:172.21.15.64:6840/146209694,v1:172.21.15.64:6841/146209694] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.h{2:24500} state up:active seq 10 addr [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.g{2:14616} state up:standby-replay seq 1 addr [v2:172.21.15.64:6838/3820392799,v1:172.21.15.64:6839/3820392799] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.c{3:24487} state up:active seq 11 addr [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.d{3:14556} state up:standby-replay seq 1 addr [v2:172.21.15.64:6836/1035088143,v1:172.21.15.64:6837/1035088143] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.f{4:24478} state up:active seq 12 addr [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.j{4:14550} state up:standby-replay seq 1 addr [v2:172.21.15.64:6834/2756017867,v1:172.21.15.64:6835/2756017867] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.703 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:45.704 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:45.704 INFO:teuthology.orchestra.run.smithi064.stdout:Standby daemons: 2023-07-14T18:24:45.704 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:45.704 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.l{-1:14526} state up:standby seq 1 addr [v2:172.21.15.144:6832/4249483747,v1:172.21.15.144:6833/4249483747] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.704 INFO:teuthology.orchestra.run.smithi064.stdout:[mds.k{-1:14538} state up:standby seq 1 addr [v2:172.21.15.105:6832/3187736260,v1:172.21.15.105:6833/3187736260] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T18:24:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 msgr2=0x7f09300a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 0x7f09300a3680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f09440ffc10 tx=0x7f094000b410 comp rx=0 tx=0).stop 2023-07-14T18:24:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 msgr2=0x7f094419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f0934006010 tx=0x7f0934004730 comp rx=0 tx=0).stop 2023-07-14T18:24:45.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 shutdown_connections 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f09440febb0 0x7f0944068c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f09300a11d0 0x7f09300a3680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0944100740 0x7f094419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 --2- 172.21.15.64:0/2399596949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09440ffdb0 0x7f094419a5e0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 >> 172.21.15.64:0/2399596949 conn(0x7f09440fa1e0 msgr2=0x7f09440fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 shutdown_connections 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:45.648+0000 7f094c143700 1 -- 172.21.15.64:0/2399596949 wait complete. 2023-07-14T18:24:45.706 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:45.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:45 smithi105 ceph-mon[98013]: pgmap v306: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 597 B/s wr, 12 op/s 2023-07-14T18:24:46.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T18:24:46.626 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:46.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:46 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2399596949' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:24:46.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:46 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2399596949' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:24:46.831 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:46 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2399596949' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T18:24:47.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:47 smithi064 ceph-mon[96720]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:47.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:47 smithi144 ceph-mon[98417]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:47.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:47 smithi105 ceph-mon[98013]: pgmap v307: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:47.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 -- 172.21.15.64:0/3116756957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1030c0 msgr2=0x7fd88c1054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:47.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 --2- 172.21.15.64:0/3116756957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1030c0 0x7fd88c1054a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd888009990 tx=0x7fd888009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 -- 172.21.15.64:0/3116756957 shutdown_connections 2023-07-14T18:24:47.917 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 --2- 172.21.15.64:0/3116756957 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 0x7fd88c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 --2- 172.21.15.64:0/3116756957 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1059e0 0x7fd88c107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 --2- 172.21.15.64:0/3116756957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1030c0 0x7fd88c1054a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 -- 172.21.15.64:0/3116756957 >> 172.21.15.64:0/3116756957 conn(0x7fd88c0fa1e0 msgr2=0x7fd88c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 -- 172.21.15.64:0/3116756957 shutdown_connections 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 -- 172.21.15.64:0/3116756957 wait complete. 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.863+0000 7fd89421d700 1 Processor -- start 2023-07-14T18:24:47.918 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 -- start start 2023-07-14T18:24:47.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 0x7fd88c1958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:47.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1030c0 0x7fd88c195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:47.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:47.919 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c19d470 con 0x7fd88c1059e0 2023-07-14T18:24:47.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c19d5b0 con 0x7fd88c1030c0 2023-07-14T18:24:47.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd89421d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88c19d740 con 0x7fd88c1007a0 2023-07-14T18:24:47.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:47.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8917b8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1030c0 0x7fd88c195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:47.920 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48458/0 (socket says 172.21.15.64:48458) 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 -- 172.21.15.64:0/2403295791 learned_addr learned my addr 172.21.15.64:0/2403295791 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd891fb9700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 0x7fd88c1958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 -- 172.21.15.64:0/2403295791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 msgr2=0x7fd88c1958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 0x7fd88c1958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 -- 172.21.15.64:0/2403295791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1030c0 msgr2=0x7fd88c195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:47.921 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1030c0 0x7fd88c195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 -- 172.21.15.64:0/2403295791 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8880095f0 con 0x7fd88c1059e0 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.864+0000 7fd8927ba700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd884007ef0 tx=0x7fd88400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.865+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884017070 con 0x7fd88c1059e0 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.865+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd88400f040 con 0x7fd88c1059e0 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.865+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd88c19da20 con 0x7fd88c1059e0 2023-07-14T18:24:47.922 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.865+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd884004020 con 0x7fd88c1059e0 2023-07-14T18:24:47.923 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.865+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88c19df20 con 0x7fd88c1059e0 2023-07-14T18:24:47.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.866+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd88c0fbd50 con 0x7fd88c1059e0 2023-07-14T18:24:47.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.867+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd88400db20 con 0x7fd88c1059e0 2023-07-14T18:24:47.924 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.867+0000 7fd87effd700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 0x7fd8780a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:47.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.867+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd8840cc150 con 0x7fd88c1059e0 2023-07-14T18:24:47.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.867+0000 7fd891fb9700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 0x7fd8780a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:47.925 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.868+0000 7fd891fb9700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 0x7fd8780a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd880009c60 tx=0x7fd880009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:47.926 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:47.872+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd8840990b0 con 0x7fd88c1059e0 2023-07-14T18:24:48.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.070+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fd88c061250 con 0x7fd88c1059e0 2023-07-14T18:24:48.122 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.071+0000 7fd87effd700 1 -- 172.21.15.64:0/2403295791 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5598 (secure 0 0 0) 0x7fd884098c40 con 0x7fd88c1059e0 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:epoch 87 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:fsid 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:created 2023-07-14T18:16:04.094382+0000 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:modified 2023-07-14T18:24:07.399732+0000 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T18:24:48.123 INFO:teuthology.orchestra.run.smithi064.stdout:crush_version 27 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:full_ratio 0.95 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:backfillfull_ratio 0.9 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:nearfull_ratio 0.85 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:require_min_compat_client luminous 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:min_compat_client luminous 2023-07-14T18:24:48.124 INFO:teuthology.orchestra.run.smithi064.stdout:require_osd_release reef 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:stretch_mode_enabled false 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:max_osd 12 2023-07-14T18:24:48.125 INFO:teuthology.orchestra.run.smithi064.stdout:osd.0 up in weight 1 up_from 8 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6802/1693019669,v1:172.21.15.64:6803/1693019669] [v2:172.21.15.64:6804/1693019669,v1:172.21.15.64:6805/1693019669] exists,up a183b85a-fd4d-4bc0-ad4f-40a7e3761093 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.1 up in weight 1 up_from 13 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6800/1803665437,v1:172.21.15.105:6801/1803665437] [v2:172.21.15.105:6802/1803665437,v1:172.21.15.105:6803/1803665437] exists,up 195865a8-38c0-48ce-afbd-447f46cc517a 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.2 up in weight 1 up_from 18 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6800/1093978507,v1:172.21.15.144:6801/1093978507] [v2:172.21.15.144:6802/1093978507,v1:172.21.15.144:6803/1093978507] exists,up c631140b-b28e-41b6-9d45-259e15845aea 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6810/2893239811,v1:172.21.15.64:6811/2893239811] [v2:172.21.15.64:6812/2893239811,v1:172.21.15.64:6813/2893239811] exists,up 0fe2d4b3-b366-4899-8979-87184b007cb8 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.4 up in weight 1 up_from 33 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6808/1784745142,v1:172.21.15.105:6809/1784745142] [v2:172.21.15.105:6810/1784745142,v1:172.21.15.105:6811/1784745142] exists,up f57e7434-c154-40a8-8b1e-13901e8bc9e7 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6808/681164415,v1:172.21.15.144:6809/681164415] [v2:172.21.15.144:6810/681164415,v1:172.21.15.144:6811/681164415] exists,up d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.6 up in weight 1 up_from 45 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6818/394567036,v1:172.21.15.64:6819/394567036] [v2:172.21.15.64:6820/394567036,v1:172.21.15.64:6821/394567036] exists,up 13faa2e5-0f41-43f8-a486-b5404b338724 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.7 up in weight 1 up_from 50 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6816/554937712,v1:172.21.15.105:6817/554937712] [v2:172.21.15.105:6818/554937712,v1:172.21.15.105:6819/554937712] exists,up cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3 2023-07-14T18:24:48.126 INFO:teuthology.orchestra.run.smithi064.stdout:osd.8 up in weight 1 up_from 57 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6816/1466260648,v1:172.21.15.144:6817/1466260648] [v2:172.21.15.144:6818/1466260648,v1:172.21.15.144:6819/1466260648] exists,up f001163e-106e-403e-94c6-81f02ae052ab 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.9 up in weight 1 up_from 64 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6826/2478630818,v1:172.21.15.64:6827/2478630818] [v2:172.21.15.64:6828/2478630818,v1:172.21.15.64:6829/2478630818] exists,up 5efb2271-1e04-4bb2-9651-52760025b9ce 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.10 up in weight 1 up_from 69 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.105:6824/1056583141,v1:172.21.15.105:6825/1056583141] [v2:172.21.15.105:6826/1056583141,v1:172.21.15.105:6827/1056583141] exists,up dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:osd.11 up in weight 1 up_from 76 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.144:6824/852142335,v1:172.21.15.144:6825/852142335] [v2:172.21.15.144:6826/852142335,v1:172.21.15.144:6827/852142335] exists,up 8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.4 [2,11] 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 2.b [8,11] 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.11 [4,8] 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.17 [4,7] 2023-07-14T18:24:48.127 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.1b [4,6] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 3.37 [5,8] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.e [9,11] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.10 [9,8] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.13 [9,8] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:pg_upmap_items 4.1d [5,10] 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/792928752 expires 2023-07-15T18:17:05.380198+0000 2023-07-14T18:24:48.128 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/792928752 expires 2023-07-15T18:17:05.380198+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/2932389061 expires 2023-07-15T18:17:05.380198+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/3783129146 expires 2023-07-15T18:16:37.487452+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1024618281 expires 2023-07-15T18:16:37.487452+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6801/3374567519 expires 2023-07-15T18:16:37.487452+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:6800/3374567519 expires 2023-07-15T18:16:37.487452+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/377975996 expires 2023-07-15T18:17:05.380198+0000 2023-07-14T18:24:48.129 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/1532744554 expires 2023-07-15T18:16:37.487452+0000 2023-07-14T18:24:48.130 INFO:teuthology.orchestra.run.smithi064.stdout:blocklist 172.21.15.64:0/2890765683 expires 2023-07-15T18:17:05.380198+0000 2023-07-14T18:24:48.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 msgr2=0x7fd8780a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 0x7fd8780a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd880009c60 tx=0x7fd880009260 comp rx=0 tx=0).stop 2023-07-14T18:24:48.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 msgr2=0x7fd88c19cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.131 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd884007ef0 tx=0x7fd88400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 shutdown_connections 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd88c1007a0 0x7fd88c1958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd8780a11d0 0x7fd8780a3680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd88c1059e0 0x7fd88c19cea0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 --2- 172.21.15.64:0/2403295791 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd88c1030c0 0x7fd88c195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.132 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 >> 172.21.15.64:0/2403295791 conn(0x7fd88c0fa1e0 msgr2=0x7fd88c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 shutdown_connections 2023-07-14T18:24:48.133 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.073+0000 7fd89421d700 1 -- 172.21.15.64:0/2403295791 wait complete. 2023-07-14T18:24:48.477 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:48 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2403295791' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:24:48.478 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T18:24:48.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:48.632 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- 172.21.15.64:0/730287692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0ffcf0 msgr2=0x7fc68c100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 --2- 172.21.15.64:0/730287692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c100140 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc678009990 tx=0x7fc678009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- 172.21.15.64:0/730287692 shutdown_connections 2023-07-14T18:24:48.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 --2- 172.21.15.64:0/730287692 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 0x7fc68c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 --2- 172.21.15.64:0/730287692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0feb00 0x7fc68c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.633 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 --2- 172.21.15.64:0/730287692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c100140 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- 172.21.15.64:0/730287692 >> 172.21.15.64:0/730287692 conn(0x7fc68c0fa100 msgr2=0x7fc68c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- 172.21.15.64:0/730287692 shutdown_connections 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- 172.21.15.64:0/730287692 wait complete. 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 Processor -- start 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.579+0000 7fc692f1c700 1 -- start start 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.634 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c194c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 0x7fc68c19bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c19c280 con 0x7fc68c0feb00 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c19c3c0 con 0x7fc68c0ffcf0 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc68c19c550 con 0x7fc68c100680 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48468/0 (socket says 172.21.15.64:48468) 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 -- 172.21.15.64:0/2738246225 learned_addr learned my addr 172.21.15.64:0/2738246225 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:48.635 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc68bfff700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c194c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc6914b9700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 0x7fc68c19bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 -- 172.21.15.64:0/2738246225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 msgr2=0x7fc68c19bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 0x7fc68c19bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 -- 172.21.15.64:0/2738246225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 msgr2=0x7fc68c194c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c194c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 -- 172.21.15.64:0/2738246225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6780095f0 con 0x7fc68c0feb00 2023-07-14T18:24:48.636 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc68bfff700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c194c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc690cb8700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc68000e970 tx=0x7fc68000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc680009790 con 0x7fc68c0feb00 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc6800044b0 con 0x7fc68c0feb00 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc68c19c830 con 0x7fc68c0feb00 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.580+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc680003dd0 con 0x7fc68c0feb00 2023-07-14T18:24:48.637 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.581+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c19cd40 con 0x7fc68c0feb00 2023-07-14T18:24:48.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.581+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc68c105a30 con 0x7fc68c0feb00 2023-07-14T18:24:48.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.582+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc680004000 con 0x7fc68c0feb00 2023-07-14T18:24:48.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.583+0000 7fc689ffb700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 0x7fc67c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.638 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.583+0000 7fc68bfff700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 0x7fc67c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.583+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc680014070 con 0x7fc68c0feb00 2023-07-14T18:24:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.584+0000 7fc68bfff700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 0x7fc67c0a3650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6780059d0 tx=0x7fc678005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:48.639 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.587+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc6800963c0 con 0x7fc68c0feb00 2023-07-14T18:24:48.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:48 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2403295791' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:24:48.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:48 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2403295791' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T18:24:48.839 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.787+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc68c02c8e0 con 0x7fc68c0feb00 2023-07-14T18:24:48.840 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.789+0000 7fc689ffb700 1 -- 172.21.15.64:0/2738246225 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7fc6800999e0 con 0x7fc68c0feb00 2023-07-14T18:24:48.841 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:48.842 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:48.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.791+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 msgr2=0x7fc67c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.791+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 0x7fc67c0a3650 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc6780059d0 tx=0x7fc678005960 comp rx=0 tx=0).stop 2023-07-14T18:24:48.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.791+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 msgr2=0x7fc68c194700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.791+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc68000e970 tx=0x7fc68000ed30 comp rx=0 tx=0).stop 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 shutdown_connections 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc68c100680 0x7fc68c19bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc67c0a11a0 0x7fc67c0a3650 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc68c0feb00 0x7fc68c194700 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 --2- 172.21.15.64:0/2738246225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc68c0ffcf0 0x7fc68c194c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 >> 172.21.15.64:0/2738246225 conn(0x7fc68c0fa100 msgr2=0x7fc68c0fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:48.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 shutdown_connections 2023-07-14T18:24:48.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.792+0000 7fc692f1c700 1 -- 172.21.15.64:0/2738246225 wait complete. 2023-07-14T18:24:48.856 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:24:48.857 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:24:48.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 -- 172.21.15.64:0/3350117180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00feb10 msgr2=0x7f63d00fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.990 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 --2- 172.21.15.64:0/3350117180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00feb10 0x7f63d00fef20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f63b8009990 tx=0x7f63b8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 -- 172.21.15.64:0/3350117180 shutdown_connections 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 --2- 172.21.15.64:0/3350117180 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 0x7f63d0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 --2- 172.21.15.64:0/3350117180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00feb10 0x7f63d00fef20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 --2- 172.21.15.64:0/3350117180 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00ffd00 0x7f63d0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 -- 172.21.15.64:0/3350117180 >> 172.21.15.64:0/3350117180 conn(0x7f63d00fa130 msgr2=0x7f63d00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 -- 172.21.15.64:0/3350117180 shutdown_connections 2023-07-14T18:24:48.991 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 -- 172.21.15.64:0/3350117180 wait complete. 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.935+0000 7f63d5b81700 1 Processor -- start 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 -- start start 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00feb10 0x7f63d0195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 0x7f63d019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.992 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d4a0 con 0x7f63d00ffd00 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d5e0 con 0x7f63d00feb10 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63d5b81700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d019d770 con 0x7f63d0100690 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48484/0 (socket says 172.21.15.64:48484) 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 -- 172.21.15.64:0/1350687551 learned_addr learned my addr 172.21.15.64:0/1350687551 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63cffff700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 0x7f63d019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63cf7fe700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00feb10 0x7f63d0195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.993 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 -- 172.21.15.64:0/1350687551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 msgr2=0x7f63d019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 0x7f63d019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 -- 172.21.15.64:0/1350687551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00feb10 msgr2=0x7f63d0195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00feb10 0x7f63d0195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.936+0000 7f63ceffd700 1 -- 172.21.15.64:0/1350687551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b80095f0 con 0x7f63d00ffd00 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63ceffd700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9a0 tx=0x7f63c000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:48.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0009790 con 0x7f63d00ffd00 2023-07-14T18:24:48.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f63c0010430 con 0x7f63d00ffd00 2023-07-14T18:24:48.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c000cea0 con 0x7f63d00ffd00 2023-07-14T18:24:48.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d019da50 con 0x7f63d00ffd00 2023-07-14T18:24:48.995 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.937+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d019df60 con 0x7f63d00ffd00 2023-07-14T18:24:48.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.938+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d010a1e0 con 0x7f63d00ffd00 2023-07-14T18:24:48.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.939+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f63c00105a0 con 0x7f63d00ffd00 2023-07-14T18:24:48.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.939+0000 7f63ccff9700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 0x7f63bc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:48.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.940+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f63c0014070 con 0x7f63d00ffd00 2023-07-14T18:24:48.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.940+0000 7f63cf7fe700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 0x7f63bc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:48.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.940+0000 7f63cf7fe700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 0x7f63bc0a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:48.997 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:48.944+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f63c0051780 con 0x7f63d00ffd00 2023-07-14T18:24:49.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.142+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63d019e570 con 0x7f63d00ffd00 2023-07-14T18:24:49.194 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.142+0000 7f63ccff9700 1 -- 172.21.15.64:0/1350687551 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63c00961f0 con 0x7f63d00ffd00 2023-07-14T18:24:49.194 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:24:49.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.144+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 msgr2=0x7f63bc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.196 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.144+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 0x7f63bc0a3680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f63b80098e0 tx=0x7f63b800b260 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.144+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 msgr2=0x7f63d0195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.144+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f63c000e9a0 tx=0x7f63c000ecb0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 shutdown_connections 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f63d0100690 0x7f63d019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f63bc0a11d0 0x7f63bc0a3680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d00feb10 0x7f63d0195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 --2- 172.21.15.64:0/1350687551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f63d00ffd00 0x7f63d0195e60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 >> 172.21.15.64:0/1350687551 conn(0x7f63d00fa130 msgr2=0x7f63d00677c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:49.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 shutdown_connections 2023-07-14T18:24:49.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.145+0000 7f63d5b81700 1 -- 172.21.15.64:0/1350687551 wait complete. 2023-07-14T18:24:49.209 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:24:49.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 -- 172.21.15.64:0/1113317032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb060100690 msgr2=0x7fb060106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 --2- 172.21.15.64:0/1113317032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb060100690 0x7fb060106f50 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb05800b0a0 tx=0x7fb05800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 -- 172.21.15.64:0/1113317032 shutdown_connections 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 --2- 172.21.15.64:0/1113317032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb060100690 0x7fb060106f50 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 --2- 172.21.15.64:0/1113317032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0600feb10 0x7fb0600fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 --2- 172.21.15.64:0/1113317032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600ffd00 0x7fb060100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 -- 172.21.15.64:0/1113317032 >> 172.21.15.64:0/1113317032 conn(0x7fb0600fa130 msgr2=0x7fb0600fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 -- 172.21.15.64:0/1113317032 shutdown_connections 2023-07-14T18:24:49.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.287+0000 7fb06808b700 1 -- 172.21.15.64:0/1113317032 wait complete. 2023-07-14T18:24:49.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 Processor -- start 2023-07-14T18:24:49.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 -- start start 2023-07-14T18:24:49.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600feb10 0x7fb060102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb060100690 0x7fb060101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060101dc0 con 0x7fb0600ffd00 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb060101f30 con 0x7fb060100690 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb06808b700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06019fa00 con 0x7fb0600feb10 2023-07-14T18:24:49.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48500/0 (socket says 172.21.15.64:48500) 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 -- 172.21.15.64:0/720477091 learned_addr learned my addr 172.21.15.64:0/720477091 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065e27700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600feb10 0x7fb060102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb066628700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb060100690 0x7fb060101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 -- 172.21.15.64:0/720477091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600feb10 msgr2=0x7fb060102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600feb10 0x7fb060102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 -- 172.21.15.64:0/720477091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb060100690 msgr2=0x7fb060101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb060100690 0x7fb060101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.288+0000 7fb065626700 1 -- 172.21.15.64:0/720477091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb058009d00 con 0x7fb0600ffd00 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb065626700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e9a0 tx=0x7fb05000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050009790 con 0x7fb0600ffd00 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb05000cbd0 con 0x7fb0600ffd00 2023-07-14T18:24:49.345 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06019fce0 con 0x7fb0600ffd00 2023-07-14T18:24:49.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050018680 con 0x7fb0600ffd00 2023-07-14T18:24:49.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.289+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0601a01e0 con 0x7fb0600ffd00 2023-07-14T18:24:49.346 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.290+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb06010a1a0 con 0x7fb0600ffd00 2023-07-14T18:24:49.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.291+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fb05000cd40 con 0x7fb0600ffd00 2023-07-14T18:24:49.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.292+0000 7fb056ffd700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 0x7fb04c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.292+0000 7fb065e27700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 0x7fb04c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.292+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fb050014070 con 0x7fb0600ffd00 2023-07-14T18:24:49.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.292+0000 7fb065e27700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 0x7fb04c0a3680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb0600ffb60 tx=0x7fb05c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:49.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.296+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb0500965c0 con 0x7fb0600ffd00 2023-07-14T18:24:49.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.495+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0601a07f0 con 0x7fb0600ffd00 2023-07-14T18:24:49.547 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.495+0000 7fb056ffd700 1 -- 172.21.15.64:0/720477091 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb0500260a0 con 0x7fb0600ffd00 2023-07-14T18:24:49.547 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:49.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 msgr2=0x7fb04c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 0x7fb04c0a3680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb0600ffb60 tx=0x7fb05c009260 comp rx=0 tx=0).stop 2023-07-14T18:24:49.550 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 msgr2=0x7fb0601010e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e9a0 tx=0x7fb05000ecb0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 shutdown_connections 2023-07-14T18:24:49.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fb0600feb10 0x7fb060102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fb04c0a11d0 0x7fb04c0a3680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb060100690 0x7fb060101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 --2- 172.21.15.64:0/720477091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0600ffd00 0x7fb0601010e0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 >> 172.21.15.64:0/720477091 conn(0x7fb0600fa130 msgr2=0x7fb0600678a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:49.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 shutdown_connections 2023-07-14T18:24:49.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.497+0000 7fb06808b700 1 -- 172.21.15.64:0/720477091 wait complete. 2023-07-14T18:24:49.560 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:49 smithi064 ceph-mon[96720]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:49.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:49 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2738246225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:49.561 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:49 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1350687551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:49.562 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:24:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:49 smithi144 ceph-mon[98417]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:49.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:49 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2738246225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:49.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:49 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1350687551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:49.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- 172.21.15.64:0/2213088719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ffd30 msgr2=0x7f981c1001a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 --2- 172.21.15.64:0/2213088719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ffd30 0x7f981c1001a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f980c009990 tx=0x7f980c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- 172.21.15.64:0/2213088719 shutdown_connections 2023-07-14T18:24:49.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 --2- 172.21.15.64:0/2213088719 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 0x7f981c106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 --2- 172.21.15.64:0/2213088719 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ff020 0x7f981c0ff430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 --2- 172.21.15.64:0/2213088719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ffd30 0x7f981c1001a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- 172.21.15.64:0/2213088719 >> 172.21.15.64:0/2213088719 conn(0x7f981c0f98a0 msgr2=0x7f981c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- 172.21.15.64:0/2213088719 shutdown_connections 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- 172.21.15.64:0/2213088719 wait complete. 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 Processor -- start 2023-07-14T18:24:49.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.640+0000 7f98247b6700 1 -- start start 2023-07-14T18:24:49.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ffd30 0x7f981c19a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 0x7f981c19cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19d370 con 0x7f981c0ff020 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19d4e0 con 0x7f981c0ffd30 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c19d6a0 con 0x7f981c1006e0 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48510/0 (socket says 172.21.15.64:48510) 2023-07-14T18:24:49.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 -- 172.21.15.64:0/2217398029 learned_addr learned my addr 172.21.15.64:0/2217398029 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9821d51700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ffd30 0x7f981c19a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822d53700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 0x7f981c19cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 -- 172.21.15.64:0/2217398029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 msgr2=0x7f981c19cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 0x7f981c19cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 -- 172.21.15.64:0/2217398029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ffd30 msgr2=0x7f981c19a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ffd30 0x7f981c19a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 -- 172.21.15.64:0/2217398029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0095f0 con 0x7f981c0ff020 2023-07-14T18:24:49.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f9822552700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f981800c990 tx=0x7f981800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818007940 con 0x7f981c0ff020 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f981800f450 con 0x7f981c0ff020 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c19d980 con 0x7f981c0ff020 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.641+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818018660 con 0x7f981c0ff020 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.642+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c19de60 con 0x7f981c0ff020 2023-07-14T18:24:49.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.642+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c105ac0 con 0x7f981c0ff020 2023-07-14T18:24:49.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.643+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9818007aa0 con 0x7f981c0ff020 2023-07-14T18:24:49.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.644+0000 7f98137fe700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:49.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.644+0000 7f9821d51700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:49.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.644+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f98180ccc70 con 0x7f981c0ff020 2023-07-14T18:24:49.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.645+0000 7f9821d51700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 0x7f98080a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f980c00b5c0 tx=0x7f980c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:49.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.647+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f98180960e0 con 0x7f981c0ff020 2023-07-14T18:24:49.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:49 smithi105 ceph-mon[98013]: pgmap v308: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 10 op/s 2023-07-14T18:24:49.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:49 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2738246225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:49.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:49 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1350687551' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:49.899 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.847+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f981c02ca70 con 0x7f981c0ff020 2023-07-14T18:24:49.900 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.848+0000 7f98137fe700 1 -- 172.21.15.64:0/2217398029 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f981800f5c0 con 0x7f981c0ff020 2023-07-14T18:24:49.900 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:24:49.901 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 msgr2=0x7f98080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 0x7f98080a3600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f980c00b5c0 tx=0x7f980c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 msgr2=0x7f981c0fe700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f981800c990 tx=0x7f981800cca0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 shutdown_connections 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f981c1006e0 0x7f981c19cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.902 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f981c0ff020 0x7f981c0fe700 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 --2- 172.21.15.64:0/2217398029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f981c0ffd30 0x7f981c19a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 >> 172.21.15.64:0/2217398029 conn(0x7f981c0f98a0 msgr2=0x7f981c0fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 shutdown_connections 2023-07-14T18:24:49.903 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.850+0000 7f98247b6700 1 -- 172.21.15.64:0/2217398029 wait complete. 2023-07-14T18:24:49.914 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:24:50.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/2807068609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00ffd00 msgr2=0x7ff2f0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/2807068609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00ffd00 0x7ff2f0100150 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009990 tx=0x7ff2dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/2807068609 shutdown_connections 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/2807068609 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f0100690 0x7ff2f0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/2807068609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f00feb10 0x7ff2f00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/2807068609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00ffd00 0x7ff2f0100150 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/2807068609 >> 172.21.15.64:0/2807068609 conn(0x7ff2f00fa130 msgr2=0x7ff2f00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/2807068609 shutdown_connections 2023-07-14T18:24:50.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.991+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/2807068609 wait complete. 2023-07-14T18:24:50.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 Processor -- start 2023-07-14T18:24:50.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 -- start start 2023-07-14T18:24:50.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f00ffd00 0x7ff2f019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 0x7ff2f019cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f019d410 con 0x7ff2f00feb10 2023-07-14T18:24:50.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f019d580 con 0x7ff2f0100690 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f6e5a700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f019d740 con 0x7ff2f00ffd00 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48524/0 (socket says 172.21.15.64:48524) 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 -- 172.21.15.64:0/3039441428 learned_addr learned my addr 172.21.15.64:0/3039441428 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2effff700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f00ffd00 0x7ff2f019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f53f7700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 0x7ff2f019cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 -- 172.21.15.64:0/3039441428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f00ffd00 msgr2=0x7ff2f019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f00ffd00 0x7ff2f019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 -- 172.21.15.64:0/3039441428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 msgr2=0x7ff2f019cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 0x7ff2f019cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f4bf6700 1 -- 172.21.15.64:0/3039441428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2dc0095f0 con 0x7ff2f00feb10 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.992+0000 7ff2f53f7700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 0x7ff2f019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:50.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2f4bf6700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400d980 tx=0x7ff2e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e400b950 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff2e4007620 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f019da20 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e400f3c0 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.993+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f019df00 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.994+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f010a190 con 0x7ff2f00feb10 2023-07-14T18:24:50.050 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.995+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff2e400bab0 con 0x7ff2f00feb10 2023-07-14T18:24:50.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.996+0000 7ff2edffb700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 0x7ff2e00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.996+0000 7ff2effff700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 0x7ff2e00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.051 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.996+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff2e40cce80 con 0x7ff2f00feb10 2023-07-14T18:24:50.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:49.996+0000 7ff2effff700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 0x7ff2e00a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c010 tx=0x7ff2dc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.052 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.000+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff2e4096260 con 0x7ff2f00feb10 2023-07-14T18:24:50.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.198+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff2f0105b10 con 0x7ff2f00feb10 2023-07-14T18:24:50.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.199+0000 7ff2edffb700 1 -- 172.21.15.64:0/3039441428 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff2e40160a0 con 0x7ff2f00feb10 2023-07-14T18:24:50.251 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:50.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 msgr2=0x7ff2e00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 0x7ff2e00a3600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc00c010 tx=0x7ff2dc005960 comp rx=0 tx=0).stop 2023-07-14T18:24:50.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 msgr2=0x7ff2f0068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff2e400d980 tx=0x7ff2e400dc90 comp rx=0 tx=0).stop 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 shutdown_connections 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff2e00a1150 0x7ff2e00a3600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff2f0100690 0x7ff2f019cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff2f00feb10 0x7ff2f0068330 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 --2- 172.21.15.64:0/3039441428 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff2f00ffd00 0x7ff2f019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 >> 172.21.15.64:0/3039441428 conn(0x7ff2f00fa130 msgr2=0x7ff2f00fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 shutdown_connections 2023-07-14T18:24:50.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.201+0000 7ff2f6e5a700 1 -- 172.21.15.64:0/3039441428 wait complete. 2023-07-14T18:24:50.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:24:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 -- 172.21.15.64:0/3570024025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 msgr2=0x7f20940fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.395 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 --2- 172.21.15.64:0/3570024025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20940fee20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f2090009990 tx=0x7f2090009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 -- 172.21.15.64:0/3570024025 shutdown_connections 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 --2- 172.21.15.64:0/3570024025 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2094100590 0x7f2094106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 --2- 172.21.15.64:0/3570024025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20940fee20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 --2- 172.21.15.64:0/3570024025 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f20940ffc00 0x7f2094100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 -- 172.21.15.64:0/3570024025 >> 172.21.15.64:0/3570024025 conn(0x7f20940fa010 msgr2=0x7f20940fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.396 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 -- 172.21.15.64:0/3570024025 shutdown_connections 2023-07-14T18:24:50.397 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.343+0000 7f209a19d700 1 -- 172.21.15.64:0/3570024025 wait complete. 2023-07-14T18:24:50.398 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 Processor -- start 2023-07-14T18:24:50.398 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 -- start start 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 0x7f20941949e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 0x7f209419ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209419c020 con 0x7f20940fea10 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209419c160 con 0x7f2094100590 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209a19d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209419c2f0 con 0x7f20940ffc00 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.399 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48538/0 (socket says 172.21.15.64:48538) 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 -- 172.21.15.64:0/831759919 learned_addr learned my addr 172.21.15.64:0/831759919 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209899a700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 0x7f20941949e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209999c700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 0x7f209419ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 -- 172.21.15.64:0/831759919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 msgr2=0x7f20941949e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 0x7f20941949e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 -- 172.21.15.64:0/831759919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 msgr2=0x7f209419ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.400 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 0x7f209419ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209919b700 1 -- 172.21.15.64:0/831759919 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20900095f0 con 0x7f20940fea10 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209999c700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 0x7f209419ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.344+0000 7f209899a700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 0x7f20941949e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f209919b700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2090000c00 tx=0x7f2090005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209001d070 con 0x7f20940fea10 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f209000be50 con 0x7f20940fea10 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209419c570 con 0x7f20940fea10 2023-07-14T18:24:50.401 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209000f460 con 0x7f20940fea10 2023-07-14T18:24:50.402 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.345+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209419c990 con 0x7f20940fea10 2023-07-14T18:24:50.402 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.346+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20941058a0 con 0x7f20940fea10 2023-07-14T18:24:50.402 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.347+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f20900226e0 con 0x7f20940fea10 2023-07-14T18:24:50.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.347+0000 7f208a7fc700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 0x7f20800a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.348+0000 7f209899a700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 0x7f20800a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.348+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f20900ce200 con 0x7f20940fea10 2023-07-14T18:24:50.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.348+0000 7f209899a700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 0x7f20800a36f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2084009be0 tx=0x7f2084009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.403 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.352+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f20900976d0 con 0x7f20940fea10 2023-07-14T18:24:50.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:50 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/720477091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:50 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2217398029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:50 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3039441428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.645 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.594+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f209402c8e0 con 0x7f20940fea10 2023-07-14T18:24:50.646 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.594+0000 7f208a7fc700 1 -- 172.21.15.64:0/831759919 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7f209009acf0 con 0x7f20940fea10 2023-07-14T18:24:50.646 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:50.647 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 msgr2=0x7f20800a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 0x7f20800a36f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2084009be0 tx=0x7f2084009260 comp rx=0 tx=0).stop 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 msgr2=0x7f20941944a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2090000c00 tx=0x7f2090005ef0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 shutdown_connections 2023-07-14T18:24:50.649 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f20800a1240 0x7f20800a36f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2094100590 0x7f209419ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f20940fea10 0x7f20941944a0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 --2- 172.21.15.64:0/831759919 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f20940ffc00 0x7f20941949e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 >> 172.21.15.64:0/831759919 conn(0x7f20940fa010 msgr2=0x7f20940fc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 shutdown_connections 2023-07-14T18:24:50.650 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.597+0000 7f209a19d700 1 -- 172.21.15.64:0/831759919 wait complete. 2023-07-14T18:24:50.651 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:50.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:50.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:50 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/720477091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:50 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2217398029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:50 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3039441428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.735+0000 7f46baae4700 1 -- 172.21.15.64:0/78046060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 msgr2=0x7f46b4100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.735+0000 7f46baae4700 1 --2- 172.21.15.64:0/78046060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4100180 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f46a0009990 tx=0x7f46a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- 172.21.15.64:0/78046060 shutdown_connections 2023-07-14T18:24:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- 172.21.15.64:0/78046060 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b41006c0 0x7f46b4106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.790 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- 172.21.15.64:0/78046060 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b40feb40 0x7f46b40fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- 172.21.15.64:0/78046060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4100180 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- 172.21.15.64:0/78046060 >> 172.21.15.64:0/78046060 conn(0x7f46b40fa160 msgr2=0x7f46b40fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- 172.21.15.64:0/78046060 shutdown_connections 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- 172.21.15.64:0/78046060 wait complete. 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 Processor -- start 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- start start 2023-07-14T18:24:50.791 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40feb40 0x7f46b41957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b41006c0 0x7f46b419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b419d330 con 0x7f46b40ffd30 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b419d470 con 0x7f46b41006c0 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46baae4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46b419d600 con 0x7f46b40feb40 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.736+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.792 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48554/0 (socket says 172.21.15.64:48554) 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 -- 172.21.15.64:0/1147456962 learned_addr learned my addr 172.21.15.64:0/1147456962 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b8880700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40feb40 0x7f46b41957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b9081700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b41006c0 0x7f46b419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 -- 172.21.15.64:0/1147456962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40feb40 msgr2=0x7f46b41957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40feb40 0x7f46b41957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 -- 172.21.15.64:0/1147456962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b41006c0 msgr2=0x7f46b419cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b41006c0 0x7f46b419cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.793 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 -- 172.21.15.64:0/1147456962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a00095f0 con 0x7f46b40ffd30 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b3fff700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f46a0005f50 tx=0x7f46a00047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a001d070 con 0x7f46b40ffd30 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f46a000be50 con 0x7f46b40ffd30 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b419d880 con 0x7f46b40ffd30 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46a0005320 con 0x7f46b40ffd30 2023-07-14T18:24:50.794 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.737+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b419dd00 con 0x7f46b40ffd30 2023-07-14T18:24:50.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.738+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b41060b0 con 0x7f46b40ffd30 2023-07-14T18:24:50.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.739+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f46a0004a80 con 0x7f46b40ffd30 2023-07-14T18:24:50.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.740+0000 7f46b1ffb700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 0x7f46a40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:50.795 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.740+0000 7f46b8880700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 0x7f46a40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:50.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.740+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f46a00cdd80 con 0x7f46b40ffd30 2023-07-14T18:24:50.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.740+0000 7f46b8880700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 0x7f46a40a36d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f46b40ffb90 tx=0x7f46a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:50.796 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.744+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f46a00971f0 con 0x7f46b40ffd30 2023-07-14T18:24:50.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:50 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/720477091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:50 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2217398029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:50 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3039441428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:50.994 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.942+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46b402c8e0 con 0x7f46b40ffd30 2023-07-14T18:24:50.996 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.944+0000 7f46b1ffb700 1 -- 172.21.15.64:0/1147456962 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7f46a009a810 con 0x7f46b40ffd30 2023-07-14T18:24:50.996 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:50.997 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 msgr2=0x7f46a40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 0x7f46a40a36d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f46b40ffb90 tx=0x7f46a8009260 comp rx=0 tx=0).stop 2023-07-14T18:24:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 msgr2=0x7f46b4195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:50.998 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f46a0005f50 tx=0x7f46a00047a0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 shutdown_connections 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f46b40feb40 0x7f46b41957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f46a40a1220 0x7f46a40a36d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46b41006c0 0x7f46b419cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 --2- 172.21.15.64:0/1147456962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f46b40ffd30 0x7f46b4195cf0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 >> 172.21.15.64:0/1147456962 conn(0x7f46b40fa160 msgr2=0x7f46b41031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:50.999 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 shutdown_connections 2023-07-14T18:24:51.000 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:50.946+0000 7f46baae4700 1 -- 172.21.15.64:0/1147456962 wait complete. 2023-07-14T18:24:51.011 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:24:51.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:24:51.141 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 -- 172.21.15.64:0/3353953701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc0ffd30 msgr2=0x7f02cc100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3353953701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc0ffd30 0x7f02cc100180 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f02bc009990 tx=0x7f02bc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 -- 172.21.15.64:0/3353953701 shutdown_connections 2023-07-14T18:24:51.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3353953701 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc1006c0 0x7f02cc106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3353953701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 0x7f02cc0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.142 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3353953701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc0ffd30 0x7f02cc100180 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 -- 172.21.15.64:0/3353953701 >> 172.21.15.64:0/3353953701 conn(0x7f02cc0fa160 msgr2=0x7f02cc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:51.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 -- 172.21.15.64:0/3353953701 shutdown_connections 2023-07-14T18:24:51.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.089+0000 7f02d3b27700 1 -- 172.21.15.64:0/3353953701 wait complete. 2023-07-14T18:24:51.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 Processor -- start 2023-07-14T18:24:51.143 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 -- start start 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 0x7f02cc195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 0x7f02cc195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d480 con 0x7f02cc1006c0 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d5c0 con 0x7f02cc0feb40 2023-07-14T18:24:51.145 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d3b27700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02cc19d750 con 0x7f02cc0ffd30 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48570/0 (socket says 172.21.15.64:48570) 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 -- 172.21.15.64:0/3826051322 learned_addr learned my addr 172.21.15.64:0/3826051322 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d18c3700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 0x7f02cc195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d10c2700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 0x7f02cc195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 -- 172.21.15.64:0/3826051322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 msgr2=0x7f02cc195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.146 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 0x7f02cc195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 -- 172.21.15.64:0/3826051322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 msgr2=0x7f02cc195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.090+0000 7f02d20c4700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 0x7f02cc195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02d20c4700 1 -- 172.21.15.64:0/3826051322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0095f0 con 0x7f02cc1006c0 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02d10c2700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 0x7f02cc195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02d20c4700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f02c400ada0 tx=0x7f02c400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c400f070 con 0x7f02cc1006c0 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f02c4007b30 con 0x7f02cc1006c0 2023-07-14T18:24:51.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02cc19da30 con 0x7f02cc1006c0 2023-07-14T18:24:51.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005280 con 0x7f02cc1006c0 2023-07-14T18:24:51.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.091+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02cc19df40 con 0x7f02cc1006c0 2023-07-14T18:24:51.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.092+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02cc10a210 con 0x7f02cc1006c0 2023-07-14T18:24:51.148 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.093+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f02c400c980 con 0x7f02cc1006c0 2023-07-14T18:24:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.093+0000 7f02c2ffd700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 0x7f02b80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.094+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f02c40cc7f0 con 0x7f02cc1006c0 2023-07-14T18:24:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.094+0000 7f02d18c3700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 0x7f02b80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.094+0000 7f02d18c3700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 0x7f02b80a3680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0ffb90 tx=0x7f02c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.149 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.097+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f02c4095cc0 con 0x7f02cc1006c0 2023-07-14T18:24:51.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.295+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f02cc19e550 con 0x7f02cc1006c0 2023-07-14T18:24:51.347 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.295+0000 7f02c2ffd700 1 -- 172.21.15.64:0/3826051322 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f02c400cc30 con 0x7f02cc1006c0 2023-07-14T18:24:51.348 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:24:51.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 msgr2=0x7f02b80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 0x7f02b80a3680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0ffb90 tx=0x7f02c800b410 comp rx=0 tx=0).stop 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 msgr2=0x7f02cc19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f02c400ada0 tx=0x7f02c400c330 comp rx=0 tx=0).stop 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 shutdown_connections 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f02b80a11d0 0x7f02b80a3680 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.350 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02cc1006c0 0x7f02cc19ceb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02cc0feb40 0x7f02cc195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 --2- 172.21.15.64:0/3826051322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f02cc0ffd30 0x7f02cc195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 >> 172.21.15.64:0/3826051322 conn(0x7f02cc0fa160 msgr2=0x7f02cc068750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:51.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 shutdown_connections 2023-07-14T18:24:51.351 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.298+0000 7f02d3b27700 1 -- 172.21.15.64:0/3826051322 wait complete. 2023-07-14T18:24:51.362 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:24:51.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 -- 172.21.15.64:0/2830239532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640ffd00 msgr2=0x7f4164100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 --2- 172.21.15.64:0/2830239532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640ffd00 0x7f4164100150 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4158009990 tx=0x7f4158009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 -- 172.21.15.64:0/2830239532 shutdown_connections 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 --2- 172.21.15.64:0/2830239532 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 0x7f4164106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 --2- 172.21.15.64:0/2830239532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640feb10 0x7f41640fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 --2- 172.21.15.64:0/2830239532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640ffd00 0x7f4164100150 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 -- 172.21.15.64:0/2830239532 >> 172.21.15.64:0/2830239532 conn(0x7f41640fa130 msgr2=0x7f41640fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:51.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 -- 172.21.15.64:0/2830239532 shutdown_connections 2023-07-14T18:24:51.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.440+0000 7f416a318700 1 -- 172.21.15.64:0/2830239532 wait complete. 2023-07-14T18:24:51.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 Processor -- start 2023-07-14T18:24:51.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 -- start start 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 0x7f416419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 0x7f416419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416419d410 con 0x7f41640feb10 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416419d580 con 0x7f41640ffd00 2023-07-14T18:24:51.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f416a318700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416419d740 con 0x7f4164100690 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f4163fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f4163fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48580/0 (socket says 172.21.15.64:48580) 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.441+0000 7f4163fff700 1 -- 172.21.15.64:0/3051869494 learned_addr learned my addr 172.21.15.64:0/3051869494 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41688b5700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 0x7f416419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41637fe700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 0x7f416419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 -- 172.21.15.64:0/3051869494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 msgr2=0x7f416419cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 0x7f416419cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 -- 172.21.15.64:0/3051869494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 msgr2=0x7f416419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 0x7f416419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 -- 172.21.15.64:0/3051869494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41580095f0 con 0x7f41640feb10 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41637fe700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 0x7f416419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f4163fff700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f415400d980 tx=0x7f415400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154004020 con 0x7f41640feb10 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4154009730 con 0x7f41640feb10 2023-07-14T18:24:51.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416419da20 con 0x7f41640feb10 2023-07-14T18:24:51.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154003a00 con 0x7f41640feb10 2023-07-14T18:24:51.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.442+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416419df30 con 0x7f41640feb10 2023-07-14T18:24:51.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.443+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416410a190 con 0x7f41640feb10 2023-07-14T18:24:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.444+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f41540098a0 con 0x7f41640feb10 2023-07-14T18:24:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.445+0000 7f41617fa700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 0x7f414c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.445+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f41540ccca0 con 0x7f41640feb10 2023-07-14T18:24:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.445+0000 7f41637fe700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 0x7f414c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.446+0000 7f41637fe700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 0x7f414c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4158000c00 tx=0x7f4158005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.448+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f41540960e0 con 0x7f41640feb10 2023-07-14T18:24:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:51 smithi144 ceph-mon[98417]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T18:24:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:51 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/831759919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:51 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1147456962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:51 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3826051322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:51.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.649+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4164105af0 con 0x7f41640feb10 2023-07-14T18:24:51.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.649+0000 7f41617fa700 1 -- 172.21.15.64:0/3051869494 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4154099700 con 0x7f41640feb10 2023-07-14T18:24:51.701 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:51.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 msgr2=0x7f414c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 0x7f414c0a3600 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4158000c00 tx=0x7f4158005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 msgr2=0x7f4164068310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f415400d980 tx=0x7f415400dc90 comp rx=0 tx=0).stop 2023-07-14T18:24:51.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 shutdown_connections 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4164100690 0x7f416419cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f414c0a1150 0x7f414c0a3600 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f41640feb10 0x7f4164068310 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 --2- 172.21.15.64:0/3051869494 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f41640ffd00 0x7f416419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 >> 172.21.15.64:0/3051869494 conn(0x7f41640fa130 msgr2=0x7f41640fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 shutdown_connections 2023-07-14T18:24:51.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.651+0000 7f416a318700 1 -- 172.21.15.64:0/3051869494 wait complete. 2023-07-14T18:24:51.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:51 smithi064 ceph-mon[96720]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T18:24:51.714 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:51 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/831759919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:51 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1147456962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.715 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:51 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3826051322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:51.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:24:51.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:51 smithi105 ceph-mon[98013]: pgmap v309: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 170 B/s wr, 10 op/s 2023-07-14T18:24:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/831759919' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1147456962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:51.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:51 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3826051322' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:24:51.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/3551394010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 msgr2=0x7fc6a80fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/3551394010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a80fef20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc698009990 tx=0x7fc698009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/3551394010 shutdown_connections 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/3551394010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 0x7fc6a8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/3551394010 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a80fef20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/3551394010 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/3551394010 >> 172.21.15.64:0/3551394010 conn(0x7fc6a80fa130 msgr2=0x7fc6a80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/3551394010 shutdown_connections 2023-07-14T18:24:51.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.793+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/3551394010 wait complete. 2023-07-14T18:24:51.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 Processor -- start 2023-07-14T18:24:51.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 -- start start 2023-07-14T18:24:51.848 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.849 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8194cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.849 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 0x7fc6a819bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.849 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a819c310 con 0x7fc6a80feb10 2023-07-14T18:24:51.849 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a819c450 con 0x7fc6a8100690 2023-07-14T18:24:51.849 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6ad8c4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a819c5e0 con 0x7fc6a80ffd00 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6a6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.794+0000 7fc6a6ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48594/0 (socket says 172.21.15.64:48594) 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 -- 172.21.15.64:0/2459168936 learned_addr learned my addr 172.21.15.64:0/2459168936 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a77fe700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 0x7fc6a819bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a67fc700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8194cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 -- 172.21.15.64:0/2459168936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 msgr2=0x7fc6a8194cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.850 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8194cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 -- 172.21.15.64:0/2459168936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 msgr2=0x7fc6a819bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 0x7fc6a819bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 -- 172.21.15.64:0/2459168936 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980095f0 con 0x7fc6a80feb10 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a67fc700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8194cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6a6ffd700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc69800b5c0 tx=0x7fc698005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69801d070 con 0x7fc6a80feb10 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a819c860 con 0x7fc6a80feb10 2023-07-14T18:24:51.851 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc698004000 con 0x7fc6a80feb10 2023-07-14T18:24:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc698022d00 con 0x7fc6a80feb10 2023-07-14T18:24:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.795+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a819cc80 con 0x7fc6a80feb10 2023-07-14T18:24:51.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.796+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a8106150 con 0x7fc6a80feb10 2023-07-14T18:24:51.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.798+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc698003b70 con 0x7fc6a80feb10 2023-07-14T18:24:51.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.798+0000 7fc6ac8c2700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 0x7fc6900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:51.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.799+0000 7fc6a67fc700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 0x7fc6900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:51.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.799+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc6980ce700 con 0x7fc6a80feb10 2023-07-14T18:24:51.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.799+0000 7fc6a67fc700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 0x7fc6900a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005fd0 tx=0x7fc69c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:51.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:51.802+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc698097bd0 con 0x7fc6a80feb10 2023-07-14T18:24:52.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.001+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6a802ca70 con 0x7fc6a80feb10 2023-07-14T18:24:52.053 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.001+0000 7fc6ac8c2700 1 -- 172.21.15.64:0/2459168936 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc698022e60 con 0x7fc6a80feb10 2023-07-14T18:24:52.054 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:24:52.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 msgr2=0x7fc6900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 0x7fc6900a3600 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc69c005fd0 tx=0x7fc69c005e30 comp rx=0 tx=0).stop 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 msgr2=0x7fc6a8194790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc69800b5c0 tx=0x7fc698005740 comp rx=0 tx=0).stop 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 shutdown_connections 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc6900a1150 0x7fc6900a3600 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc6a8100690 0x7fc6a819bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6a80feb10 0x7fc6a8194790 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 --2- 172.21.15.64:0/2459168936 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc6a80ffd00 0x7fc6a8194cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 >> 172.21.15.64:0/2459168936 conn(0x7fc6a80fa130 msgr2=0x7fc6a8103290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 shutdown_connections 2023-07-14T18:24:52.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.004+0000 7fc6ad8c4700 1 -- 172.21.15.64:0/2459168936 wait complete. 2023-07-14T18:24:52.068 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:24:52.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 -- 172.21.15.64:0/2616892529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b00100e00 msgr2=0x7f7b00101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.197 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 --2- 172.21.15.64:0/2616892529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b00100e00 0x7f7b00101250 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7af0009990 tx=0x7f7af0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 -- 172.21.15.64:0/2616892529 shutdown_connections 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 --2- 172.21.15.64:0/2616892529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 0x7f7b00108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 --2- 172.21.15.64:0/2616892529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b000ffc10 0x7f7b00100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 --2- 172.21.15.64:0/2616892529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b00100e00 0x7f7b00101250 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 -- 172.21.15.64:0/2616892529 >> 172.21.15.64:0/2616892529 conn(0x7f7b000fb250 msgr2=0x7f7b000fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.198 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 -- 172.21.15.64:0/2616892529 shutdown_connections 2023-07-14T18:24:52.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 -- 172.21.15.64:0/2616892529 wait complete. 2023-07-14T18:24:52.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.145+0000 7f7b06a18700 1 Processor -- start 2023-07-14T18:24:52.199 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 -- start start 2023-07-14T18:24:52.201 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 0x7f7b00195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 0x7f7b0019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0019d3e0 con 0x7f7b000ffc10 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0019d520 con 0x7f7b00100e00 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b06a18700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0019d6b0 con 0x7f7b00101790 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.202 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48608/0 (socket says 172.21.15.64:48608) 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 -- 172.21.15.64:0/899996034 learned_addr learned my addr 172.21.15.64:0/899996034 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7b04fb5700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 0x7f7b0019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7aff7fe700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 0x7f7b00195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 -- 172.21.15.64:0/899996034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 msgr2=0x7f7b0019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 0x7f7b0019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.203 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 -- 172.21.15.64:0/899996034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 msgr2=0x7f7b00195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 0x7f7b00195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7afffff700 1 -- 172.21.15.64:0/899996034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00095f0 con 0x7f7b000ffc10 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.146+0000 7f7aff7fe700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 0x7f7b00195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7afffff700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e970 tx=0x7f7ae800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8009790 con 0x7f7b000ffc10 2023-07-14T18:24:52.204 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ae80044b0 con 0x7f7b000ffc10 2023-07-14T18:24:52.205 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0019d990 con 0x7f7b000ffc10 2023-07-14T18:24:52.205 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae8003dd0 con 0x7f7b000ffc10 2023-07-14T18:24:52.205 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.147+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0019dea0 con 0x7f7b000ffc10 2023-07-14T18:24:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.148+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0010b290 con 0x7f7b000ffc10 2023-07-14T18:24:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.149+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7ae8004000 con 0x7f7b000ffc10 2023-07-14T18:24:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.150+0000 7f7afd7fa700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 0x7f7aec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.150+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7ae8014070 con 0x7f7b000ffc10 2023-07-14T18:24:52.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.150+0000 7f7aff7fe700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 0x7f7aec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.150+0000 7f7aff7fe700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 0x7f7aec0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b2e0 tx=0x7f7af0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.153+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ae8096470 con 0x7f7b000ffc10 2023-07-14T18:24:52.405 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.354+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7b00106b90 con 0x7f7b000ffc10 2023-07-14T18:24:52.406 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.354+0000 7f7afd7fa700 1 -- 172.21.15.64:0/899996034 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7ae8099a90 con 0x7f7b000ffc10 2023-07-14T18:24:52.406 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:24:52.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 msgr2=0x7f7aec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.408 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 0x7f7aec0a3600 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7af000b2e0 tx=0x7f7af0005fb0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 msgr2=0x7f7b00195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f7ae800e970 tx=0x7f7ae800ed30 comp rx=0 tx=0).stop 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 shutdown_connections 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f7b00101790 0x7f7b0019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f7aec0a1150 0x7f7aec0a3600 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b000ffc10 0x7f7b00195860 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.409 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 --2- 172.21.15.64:0/899996034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b00100e00 0x7f7b00195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.356+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 >> 172.21.15.64:0/899996034 conn(0x7f7b000fb250 msgr2=0x7f7b000fd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.357+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 shutdown_connections 2023-07-14T18:24:52.410 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.357+0000 7f7b06a18700 1 -- 172.21.15.64:0/899996034 wait complete. 2023-07-14T18:24:52.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:24:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 -- 172.21.15.64:0/1965132975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 msgr2=0x7fd4a8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/1965132975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a8107d20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd4a000b0a0 tx=0x7fd4a000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 -- 172.21.15.64:0/1965132975 shutdown_connections 2023-07-14T18:24:52.548 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/1965132975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a8107d20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/1965132975 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a8102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/1965132975 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 0x7fd4a81053c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.495+0000 7fd4ad0df700 1 -- 172.21.15.64:0/1965132975 >> 172.21.15.64:0/1965132975 conn(0x7fd4a80fa100 msgr2=0x7fd4a80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.496+0000 7fd4ad0df700 1 -- 172.21.15.64:0/1965132975 shutdown_connections 2023-07-14T18:24:52.549 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.496+0000 7fd4ad0df700 1 -- 172.21.15.64:0/1965132975 wait complete. 2023-07-14T18:24:52.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.496+0000 7fd4ad0df700 1 Processor -- start 2023-07-14T18:24:52.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 -- start start 2023-07-14T18:24:52.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a81958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 0x7fd4a8195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.551 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819d460 con 0x7fd4a8105900 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819d5a0 con 0x7fd4a81006c0 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4ad0df700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a819d730 con 0x7fd4a8102fe0 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48626/0 (socket says 172.21.15.64:48626) 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 -- 172.21.15.64:0/900441260 learned_addr learned my addr 172.21.15.64:0/900441260 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:52.552 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a6d9d700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a81958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a659c700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 0x7fd4a8195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 -- 172.21.15.64:0/900441260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 msgr2=0x7fd4a8195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 0x7fd4a8195e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 -- 172.21.15.64:0/900441260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 msgr2=0x7fd4a81958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a81958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a759e700 1 -- 172.21.15.64:0/900441260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a0009d00 con 0x7fd4a8105900 2023-07-14T18:24:52.553 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.497+0000 7fd4a6d9d700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a81958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd4a759e700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00062a0 tx=0x7fd4a0004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a000e040 con 0x7fd4a8105900 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4a00094c0 con 0x7fd4a8105900 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a819d9b0 con 0x7fd4a8105900 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4a001bc70 con 0x7fd4a8105900 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.498+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a819dde0 con 0x7fd4a8105900 2023-07-14T18:24:52.554 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.499+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a80fbc70 con 0x7fd4a8105900 2023-07-14T18:24:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.500+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd4a001d890 con 0x7fd4a8105900 2023-07-14T18:24:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.501+0000 7fd48ffff700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 0x7fd4900a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.501+0000 7fd4a6d9d700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 0x7fd4900a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.555 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.501+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd4a00cf210 con 0x7fd4a8105900 2023-07-14T18:24:52.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.501+0000 7fd4a6d9d700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 0x7fd4900a3650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd498005950 tx=0x7fd498005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.556 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.504+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4a00987a0 con 0x7fd4a8105900 2023-07-14T18:24:52.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:52 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3051869494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:52 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2459168936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.799 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.747+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4a802c8e0 con 0x7fd4a8105900 2023-07-14T18:24:52.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.748+0000 7fd48ffff700 1 -- 172.21.15.64:0/900441260 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+10891 (secure 0 0 0) 0x7fd4a009bdc0 con 0x7fd4a8105900 2023-07-14T18:24:52.800 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:52.800 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:23:28.278497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:24:52.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 msgr2=0x7fd4900a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 0x7fd4900a3650 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd498005950 tx=0x7fd498005f50 comp rx=0 tx=0).stop 2023-07-14T18:24:52.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 msgr2=0x7fd4a819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd4a00062a0 tx=0x7fd4a0004db0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 shutdown_connections 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd4900a11a0 0x7fd4900a3650 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4a8105900 0x7fd4a819ce90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a81006c0 0x7fd4a81958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 --2- 172.21.15.64:0/900441260 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fd4a8102fe0 0x7fd4a8195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 >> 172.21.15.64:0/900441260 conn(0x7fd4a80fa100 msgr2=0x7fd4a8101320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 shutdown_connections 2023-07-14T18:24:52.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.750+0000 7fd4ad0df700 1 -- 172.21.15.64:0/900441260 wait complete. 2023-07-14T18:24:52.804 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 27 2023-07-14T18:24:52.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:52 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3051869494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:52 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2459168936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.813 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:52 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3051869494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.813 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:52 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2459168936' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:24:52.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:24:52.940 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 -- 172.21.15.64:0/3192539370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 msgr2=0x7f3160106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 --2- 172.21.15.64:0/3192539370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f3160106f40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f315800b0a0 tx=0x7f315800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 -- 172.21.15.64:0/3192539370 shutdown_connections 2023-07-14T18:24:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 --2- 172.21.15.64:0/3192539370 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600feb00 0x7f31600fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 --2- 172.21.15.64:0/3192539370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f3160106f40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 --2- 172.21.15.64:0/3192539370 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600ffcf0 0x7f3160100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 -- 172.21.15.64:0/3192539370 >> 172.21.15.64:0/3192539370 conn(0x7f31600fa100 msgr2=0x7f31600fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:52.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 -- 172.21.15.64:0/3192539370 shutdown_connections 2023-07-14T18:24:52.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.888+0000 7f3164f89700 1 -- 172.21.15.64:0/3192539370 wait complete. 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 Processor -- start 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 -- start start 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600feb00 0x7f3160195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600ffcf0 0x7f3160195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.944 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316019d3c0 con 0x7f3160100680 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316019d500 con 0x7f31600feb00 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.889+0000 7f3164f89700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f316019d690 con 0x7f31600ffcf0 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48640/0 (socket says 172.21.15.64:48640) 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 -- 172.21.15.64:0/899694032 learned_addr learned my addr 172.21.15.64:0/899694032 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315e59c700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600feb00 0x7f3160195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.945 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315dd9b700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600ffcf0 0x7f3160195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 -- 172.21.15.64:0/899694032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600ffcf0 msgr2=0x7f3160195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600ffcf0 0x7f3160195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 -- 172.21.15.64:0/899694032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600feb00 msgr2=0x7f3160195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600feb00 0x7f3160195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 -- 172.21.15.64:0/899694032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3158009d00 con 0x7f3160100680 2023-07-14T18:24:52.946 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f315ed9d700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f315800bad0 tx=0x7f315800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315800e040 con 0x7f3160100680 2023-07-14T18:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3158004e40 con 0x7f3160100680 2023-07-14T18:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f316019d910 con 0x7f3160100680 2023-07-14T18:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580052a0 con 0x7f3160100680 2023-07-14T18:24:52.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.890+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f316019dde0 con 0x7f3160100680 2023-07-14T18:24:52.948 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.891+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f316010a190 con 0x7f3160100680 2023-07-14T18:24:52.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.893+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f315801d990 con 0x7f3160100680 2023-07-14T18:24:52.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.893+0000 7f314f7fe700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 0x7f31480a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:52.949 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.894+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31580ce5e0 con 0x7f3160100680 2023-07-14T18:24:52.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.894+0000 7f315e59c700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 0x7f31480a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:52.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.894+0000 7f315e59c700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 0x7f31480a36d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31600ffb50 tx=0x7f315000a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:52.950 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:52.897+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3158097ab0 con 0x7f3160100680 2023-07-14T18:24:53.147 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.096+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f316019e3c0 con 0x7f3160100680 2023-07-14T18:24:53.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.097+0000 7f314f7fe700 1 -- 172.21.15.64:0/899694032 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7f315809b0d0 con 0x7f3160100680 2023-07-14T18:24:53.151 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:24:53.151 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:24:53.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.099+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 msgr2=0x7f31480a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.099+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 0x7f31480a36d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31600ffb50 tx=0x7f315000a3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.099+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 msgr2=0x7f316019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.099+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f315800bad0 tx=0x7f315800bb00 comp rx=0 tx=0).stop 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 shutdown_connections 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31480a1220 0x7f31480a36d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3160100680 0x7f316019cdf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31600feb00 0x7f3160195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 --2- 172.21.15.64:0/899694032 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31600ffcf0 0x7f3160195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 >> 172.21.15.64:0/899694032 conn(0x7f31600fa100 msgr2=0x7f31600677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 shutdown_connections 2023-07-14T18:24:53.154 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.100+0000 7f3164f89700 1 -- 172.21.15.64:0/899694032 wait complete. 2023-07-14T18:24:53.164 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T18:23:04.865157+0000', 'modified': '2023-07-14T18:23:28.278497+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24545, 'mds_1': 24508, 'mds_2': 24500, 'mds_3': 24487, 'mds_4': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14550': {'gid': 14550, 'name': 'j', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/2756017867', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 2756017867}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 2756017867}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14556': {'gid': 14556, 'name': 'd', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6837/1035088143', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 1035088143}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 1035088143}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14616': {'gid': 14616, 'name': 'g', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6839/3820392799', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 3820392799}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 3820392799}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_14628': {'gid': 14628, 'name': 'a', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6841/146209694', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6840', 'nonce': 146209694}, {'type': 'v1', 'addr': '172.21.15.64:6841', 'nonce': 146209694}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.105:6835/2365162845', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6834', 'nonce': 2365162845}, {'type': 'v1', 'addr': '172.21.15.105:6835', 'nonce': 2365162845}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.144:6835/4223527896', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6834', 'nonce': 4223527896}, {'type': 'v1', 'addr': '172.21.15.144:6835', 'nonce': 4223527896}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24487': {'gid': 24487, 'name': 'c', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.144:6837/1582348642', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6836', 'nonce': 1582348642}, {'type': 'v1', 'addr': '172.21.15.144:6837', 'nonce': 1582348642}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24500': {'gid': 24500, 'name': 'h', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.105:6837/2898950423', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6836', 'nonce': 2898950423}, {'type': 'v1', 'addr': '172.21.15.105:6837', 'nonce': 2898950423}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24508': {'gid': 24508, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.144:6839/4051776603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.144:6838', 'nonce': 4051776603}, {'type': 'v1', 'addr': '172.21.15.144:6839', 'nonce': 4051776603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24545': {'gid': 24545, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.105:6839/2402376208', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.105:6838', 'nonce': 2402376208}, {'type': 'v1', 'addr': '172.21.15.105:6839', 'nonce': 2402376208}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T18:24:53.165 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2023-07-14T18:24:53.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-07-14T18:24:53.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 -- 172.21.15.64:0/1882994385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47281006a0 msgr2=0x7f4728106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.294 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 --2- 172.21.15.64:0/1882994385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47281006a0 0x7f4728106f60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f472000b0a0 tx=0x7f472000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 -- 172.21.15.64:0/1882994385 shutdown_connections 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 --2- 172.21.15.64:0/1882994385 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280feb20 0x7f47280fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 --2- 172.21.15.64:0/1882994385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47281006a0 0x7f4728106f60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 --2- 172.21.15.64:0/1882994385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47280ffd10 0x7f4728100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 -- 172.21.15.64:0/1882994385 >> 172.21.15.64:0/1882994385 conn(0x7f47280fa160 msgr2=0x7f47280fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 -- 172.21.15.64:0/1882994385 shutdown_connections 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.240+0000 7f472eb68700 1 -- 172.21.15.64:0/1882994385 wait complete. 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 Processor -- start 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 -- start start 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280ffd10 0x7f47281013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47281006a0 0x7f4728101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47281020b0 con 0x7f47280feb20 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4728102220 con 0x7f47281006a0 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472eb68700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472819f9c0 con 0x7f47280ffd10 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48650/0 (socket says 172.21.15.64:48650) 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 -- 172.21.15.64:0/887576940 learned_addr learned my addr 172.21.15.64:0/887576940 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472d105700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47281006a0 0x7f4728101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f4727fff700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280ffd10 0x7f47281013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 -- 172.21.15.64:0/887576940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280ffd10 msgr2=0x7f47281013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280ffd10 0x7f47281013d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 -- 172.21.15.64:0/887576940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47281006a0 msgr2=0x7f4728101910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47281006a0 0x7f4728101910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.241+0000 7f472c904700 1 -- 172.21.15.64:0/887576940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4720009d00 con 0x7f47280feb20 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f472c904700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f471c00d980 tx=0x7f471c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.298 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f4725ffb700 1 -- 172.21.15.64:0/887576940 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c004020 con 0x7f47280feb20 2023-07-14T18:24:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f4725ffb700 1 -- 172.21.15.64:0/887576940 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f471c009730 con 0x7f47280feb20 2023-07-14T18:24:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f4725ffb700 1 -- 172.21.15.64:0/887576940 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c003a00 con 0x7f47280feb20 2023-07-14T18:24:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472819fca0 con 0x7f47280feb20 2023-07-14T18:24:53.299 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.242+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47281a01b0 con 0x7f47280feb20 2023-07-14T18:24:53.304 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.244+0000 7f4725ffb700 1 -- 172.21.15.64:0/887576940 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f471c0098a0 con 0x7f47280feb20 2023-07-14T18:24:53.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.244+0000 7f4725ffb700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 0x7f47180a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.245+0000 7f4725ffb700 1 -- 172.21.15.64:0/887576940 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f471c0cc890 con 0x7f47280feb20 2023-07-14T18:24:53.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.245+0000 7f4727fff700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 0x7f47180a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.305 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.245+0000 7f4727fff700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 0x7f47180a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4714009b30 tx=0x7f4714009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 Processor -- start 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 -- start start 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5b6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5b78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c4bac60 con 0x55e31c5b5f10 2023-07-14T18:24:53.306 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c50d9f0 con 0x55e31c5b6860 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f473ad00b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c506e80 con 0x55e31c5b7460 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48658/0 (socket says 172.21.15.64:48658) 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 -- 172.21.15.64:0/1194262411 learned_addr learned my addr 172.21.15.64:0/1194262411 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f4727fff700 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5b6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472d105700 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5b78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.307 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 -- 172.21.15.64:0/1194262411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 msgr2=0x55e31c5b78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5b78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 -- 172.21.15.64:0/1194262411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 msgr2=0x55e31c5b6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5b6e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.246+0000 7f472c904700 1 -- 172.21.15.64:0/1194262411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e31c334d90 con 0x55e31c5b5f10 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f472c904700 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f471c0a18b0 tx=0x7f471c099890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5619295dc3726e16 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f4712ffd700 1 -- 172.21.15.64:0/1194262411 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c009b80 con 0x55e31c5b5f10 2023-07-14T18:24:53.308 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f4712ffd700 1 -- 172.21.15.64:0/1194262411 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f471c099990 con 0x55e31c5b5f10 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f4712ffd700 1 -- 172.21.15.64:0/1194262411 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c0990c0 con 0x55e31c5b5f10 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 -- 172.21.15.64:0/1194262411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 msgr2=0x55e31c5b6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f471c0a18b0 tx=0x7f471c099890 comp rx=0 tx=0).stop 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 -- 172.21.15.64:0/1194262411 shutdown_connections 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5b78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.309 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5b6320 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1194262411 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5b6e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.247+0000 7f473ad00b80 1 -- 172.21.15.64:0/1194262411 >> 172.21.15.64:0/1194262411 conn(0x55e31c5ab630 msgr2=0x55e31c5b0980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- 172.21.15.64:0/1194262411 shutdown_connections 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- 172.21.15.64:0/1194262411 wait complete. 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- start start 2023-07-14T18:24:53.310 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 Processor -- start 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5c6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c4a6a70 con 0x55e31c5b5f10 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c4a6770 con 0x55e31c5b6860 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c38a8d0 con 0x55e31c5b7460 2023-07-14T18:24:53.311 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f473ad00b80 10 client.0 _ll_register_callbacks cb 0x55e31c4a4710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f472c904700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48664/0 (socket says 172.21.15.64:48664) 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.248+0000 7f472c904700 1 -- 172.21.15.64:0/1555413723 learned_addr learned my addr 172.21.15.64:0/1555413723 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4727fff700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472d105700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5c6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 -- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 msgr2=0x55e31c5c6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5c6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 -- 172.21.15.64:0/1555413723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 msgr2=0x55e31c5bfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5bfef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 -- 172.21.15.64:0/1555413723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47281a01b0 con 0x55e31c5b5f10 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472d105700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f472c904700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f471c0ca0d0 tx=0x7f471c095d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfe3060ea3d6678 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4707fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f471c0b21a0 con 0x55e31c5b5f10 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f471c098e80 con 0x55e31c5b5f10 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e31c4604a0 con 0x55e31c5b5f10 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f471c099040 con 0x55e31c5b5f10 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f473ad00b80 10 client.15441 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e31c31dc10 con 0x55e31c5b5f10 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.249+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55e31c4604a0 con 0x55e31c5b5f10 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f471c0c7070 con 0x55e31c5b5f10 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 10 client.15441 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 10 client.15441 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 20 client.15441 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c5caf60 0x55e31c5c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 4 client.15441 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 --> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e31c584d50 con 0x55e31c5caf60 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f4727fff700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c5caf60 0x55e31c5c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f4727fff700 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c5caf60 0x55e31c5c9e60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=92a9f9b85fedfad1 server_cookie=76053b25cac94c4a in_seq=0 out_seq=0 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.250+0000 7f4707fff700 10 client.15441 ms_handle_connect on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f4707fff700 1 -- 172.21.15.64:0/1555413723 <== mds.4 v2:172.21.15.144:6834/4223527896 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55e31c584d50 con 0x55e31c5caf60 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f4707fff700 10 client.15441 handle_command_reply: tid=0 2023-07-14T18:24:53.316 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 client.15441 shutdown 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 msgr2=0x55e31c5bf9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f471c0ca0d0 tx=0x7f471c095d50 comp rx=0 tx=0).stop 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 shutdown_connections 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c5caf60 0x55e31c5c9e60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b7460 0x55e31c5c6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b5f10 0x55e31c5bf9b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1555413723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5bfef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f4707fff700 0 client.15441 ms_handle_reset on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:53.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 >> 172.21.15.64:0/1555413723 conn(0x55e31c5ab630 msgr2=0x55e31c5af600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 shutdown_connections 2023-07-14T18:24:53.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 1 -- 172.21.15.64:0/1555413723 wait complete. 2023-07-14T18:24:53.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.251+0000 7f473ad00b80 20 client.15441 trim_cache size 0 max 16384 2023-07-14T18:24:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 Processor -- start 2023-07-14T18:24:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 -- start start 2023-07-14T18:24:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b5f10 0x55e31c5be620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b6860 0x55e31c5c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c5b7d50 con 0x55e31c5b7460 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c584d50 con 0x55e31c5b5f10 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f473ad00b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c4604a0 con 0x55e31c5b6860 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f472d105700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.269+0000 7f472d105700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48676/0 (socket says 172.21.15.64:48676) 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 -- 172.21.15.64:0/1627659351 learned_addr learned my addr 172.21.15.64:0/1627659351 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472c904700 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b5f10 0x55e31c5be620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f4727fff700 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b6860 0x55e31c5c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 -- 172.21.15.64:0/1627659351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b6860 msgr2=0x55e31c5c6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b6860 0x55e31c5c6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 -- 172.21.15.64:0/1627659351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b5f10 msgr2=0x55e31c5be620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b5f10 0x55e31c5be620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 -- 172.21.15.64:0/1627659351 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e31c35b200 con 0x55e31c5b7460 2023-07-14T18:24:53.324 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f472d105700 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f47200093b0 tx=0x7f47200093e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1342e80ece4373f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f47137fe700 1 -- 172.21.15.64:0/1627659351 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000e040 con 0x55e31c5b7460 2023-07-14T18:24:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f47137fe700 1 -- 172.21.15.64:0/1627659351 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47200076e0 con 0x55e31c5b7460 2023-07-14T18:24:53.325 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.270+0000 7f47137fe700 1 -- 172.21.15.64:0/1627659351 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4720003de0 con 0x55e31c5b7460 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- 172.21.15.64:0/1627659351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 msgr2=0x55e31c5b00a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f47200093b0 tx=0x7f47200093e0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- 172.21.15.64:0/1627659351 shutdown_connections 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5b00a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b5f10 0x55e31c5be620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.329 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 --2- 172.21.15.64:0/1627659351 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b6860 0x55e31c5c6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- 172.21.15.64:0/1627659351 >> 172.21.15.64:0/1627659351 conn(0x55e31c5ab630 msgr2=0x55e31c5acb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- 172.21.15.64:0/1627659351 shutdown_connections 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- 172.21.15.64:0/1627659351 wait complete. 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 -- start start 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.271+0000 7f473ad00b80 1 Processor -- start 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 0x55e31c5c4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.330 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5c4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c5c5ba0 con 0x55e31c5b7460 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c5c5d10 con 0x55e31c5b6860 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e31c5c5ed0 con 0x55e31c5b5f10 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 10 client.0 _ll_register_callbacks cb 0x7f471c0994d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.331 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48682/0 (socket says 172.21.15.64:48682) 2023-07-14T18:24:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 -- 172.21.15.64:0/2151851307 learned_addr learned my addr 172.21.15.64:0/2151851307 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f4727fff700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5c4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.332 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472c904700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 0x55e31c5c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 -- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 msgr2=0x55e31c5c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 0x55e31c5c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 -- 172.21.15.64:0/2151851307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 msgr2=0x55e31c5c4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.333 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5c4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 -- 172.21.15.64:0/2151851307 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4728107890 con 0x55e31c5b7460 2023-07-14T18:24:53.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472c904700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 0x55e31c5c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:53.334 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f472d105700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f472000beb0 tx=0x7f4720007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4c8163e0564ec8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f4707fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472000e040 con 0x55e31c5b7460 2023-07-14T18:24:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4720003ce0 con 0x55e31c5b7460 2023-07-14T18:24:53.335 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.272+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f471c099990 con 0x55e31c5b7460 2023-07-14T18:24:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4720003ea0 con 0x55e31c5b7460 2023-07-14T18:24:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 10 client.15447 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.336 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e31c333260 con 0x55e31c5b7460 2023-07-14T18:24:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f471c099990 con 0x55e31c5b7460 2023-07-14T18:24:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4720023030 con 0x55e31c5b7460 2023-07-14T18:24:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 10 client.15447 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.337 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 10 client.15447 resolve_mds: resolved name 'f' to daemon mds.f 2023-07-14T18:24:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 20 client.15447 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.338 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c623a20 0x55e31c6205e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 4 client.15447 mds_command: new command op to 24478 tid=0[{"prefix": "status"}] 2023-07-14T18:24:53.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 --> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e31c4ffe60 con 0x55e31c623a20 2023-07-14T18:24:53.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.273+0000 7f472c904700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c623a20 0x55e31c6205e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.339 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f472c904700 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c623a20 0x55e31c6205e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b345c7258716866b server_cookie=614f228a1f8e45f3 in_seq=0 out_seq=0 2023-07-14T18:24:53.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f4707fff700 10 client.15447 ms_handle_connect on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:53.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f4707fff700 1 -- 172.21.15.64:0/2151851307 <== mds.4 v2:172.21.15.144:6834/4223527896 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e31c4ffe60 con 0x55e31c623a20 2023-07-14T18:24:53.340 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f4707fff700 10 client.15447 handle_command_reply: tid=0 2023-07-14T18:24:53.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 client.15447 shutdown 2023-07-14T18:24:53.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 msgr2=0x55e31c5c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.341 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f472000beb0 tx=0x7f4720007e80 comp rx=0 tx=0).stop 2023-07-14T18:24:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 shutdown_connections 2023-07-14T18:24:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:6834/4223527896,v1:172.21.15.144:6835/4223527896] conn(0x55e31c623a20 0x55e31c6205e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55e31c5b5f10 0x55e31c5c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e31c5b7460 0x55e31c5c5660 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.342 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f4707fff700 0 client.15447 ms_handle_reset on v2:172.21.15.144:6834/4223527896 2023-07-14T18:24:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 --2- 172.21.15.64:0/2151851307 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55e31c5b6860 0x55e31c5c4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 >> 172.21.15.64:0/2151851307 conn(0x55e31c5ab630 msgr2=0x55e31c5be620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 shutdown_connections 2023-07-14T18:24:53.343 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 1 -- 172.21.15.64:0/2151851307 wait complete. 2023-07-14T18:24:53.344 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.274+0000 7f473ad00b80 20 client.15447 trim_cache size 0 max 16384 2023-07-14T18:24:53.344 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:53.345 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:53.345 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 4, 2023-07-14T18:24:53.345 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24478, 2023-07-14T18:24:53.345 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:53.345 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:53.346 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:53.346 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 86.014052333999999, 2023-07-14T18:24:53.346 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:53.346 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:53.346 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:53.347 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 127.551571935 2023-07-14T18:24:53.347 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 msgr2=0x7f47180a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 0x7f47180a3680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4714009b30 tx=0x7f4714009260 comp rx=0 tx=0).stop 2023-07-14T18:24:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 msgr2=0x7f4728102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f471c00d980 tx=0x7f471c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:24:53.348 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 shutdown_connections 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f47180a11d0 0x7f47180a3680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47281006a0 0x7f4728101910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47280feb20 0x7f4728102d90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 --2- 172.21.15.64:0/887576940 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f47280ffd10 0x7f47281013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 >> 172.21.15.64:0/887576940 conn(0x7f47280fa160 msgr2=0x7f47280fc340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 shutdown_connections 2023-07-14T18:24:53.349 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.275+0000 7f472eb68700 1 -- 172.21.15.64:0/887576940 wait complete. 2023-07-14T18:24:53.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-07-14T18:24:53.480 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 -- 172.21.15.64:0/3898183890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0ffd30 msgr2=0x7f31cc100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 --2- 172.21.15.64:0/3898183890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0ffd30 0x7f31cc100180 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f31b8009990 tx=0x7f31b8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 -- 172.21.15.64:0/3898183890 shutdown_connections 2023-07-14T18:24:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 --2- 172.21.15.64:0/3898183890 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 0x7f31cc106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.481 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 --2- 172.21.15.64:0/3898183890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0feb40 0x7f31cc0fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 --2- 172.21.15.64:0/3898183890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0ffd30 0x7f31cc100180 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 -- 172.21.15.64:0/3898183890 >> 172.21.15.64:0/3898183890 conn(0x7f31cc0fa160 msgr2=0x7f31cc0fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 -- 172.21.15.64:0/3898183890 shutdown_connections 2023-07-14T18:24:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.426+0000 7f31d30a8700 1 -- 172.21.15.64:0/3898183890 wait complete. 2023-07-14T18:24:53.482 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 Processor -- start 2023-07-14T18:24:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 -- start start 2023-07-14T18:24:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0ffd30 0x7f31cc194ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 0x7f31cc19bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.483 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc19c320 con 0x7f31cc0feb40 2023-07-14T18:24:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc19c460 con 0x7f31cc0ffd30 2023-07-14T18:24:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d30a8700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31cc19c5f0 con 0x7f31cc1006c0 2023-07-14T18:24:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48692/0 (socket says 172.21.15.64:48692) 2023-07-14T18:24:53.484 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 -- 172.21.15.64:0/1910617737 learned_addr learned my addr 172.21.15.64:0/1910617737 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0ffd30 0x7f31cc194ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 -- 172.21.15.64:0/1910617737 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 msgr2=0x7f31cc19bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d1645700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 0x7f31cc19bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 0x7f31cc19bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.485 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 -- 172.21.15.64:0/1910617737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0ffd30 msgr2=0x7f31cc194ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0ffd30 0x7f31cc194ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.427+0000 7f31d0e44700 1 -- 172.21.15.64:0/1910617737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80095f0 con 0x7f31cc0feb40 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f31c000ea80 tx=0x7f31c000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31c9ffb700 1 -- 172.21.15.64:0/1910617737 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0013070 con 0x7f31cc0feb40 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31c9ffb700 1 -- 172.21.15.64:0/1910617737 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31c000be50 con 0x7f31cc0feb40 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31c9ffb700 1 -- 172.21.15.64:0/1910617737 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c0003a00 con 0x7f31cc0feb40 2023-07-14T18:24:53.486 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31cc0687d0 con 0x7f31cc0feb40 2023-07-14T18:24:53.487 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.428+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc068ce0 con 0x7f31cc0feb40 2023-07-14T18:24:53.491 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.431+0000 7f31c9ffb700 1 -- 172.21.15.64:0/1910617737 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f31c000b9c0 con 0x7f31cc0feb40 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.431+0000 7f31c9ffb700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 0x7f31bc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.431+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 0x7f31bc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.431+0000 7f31c9ffb700 1 -- 172.21.15.64:0/1910617737 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31c00ce2c0 con 0x7f31cc0feb40 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.432+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 0x7f31bc0a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b2e0 tx=0x7f31b8004000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.432+0000 7f31df240b80 1 Processor -- start 2023-07-14T18:24:53.492 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.432+0000 7f31df240b80 1 -- start start 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acdf10 0x556557ace320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557acf8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565579d2c60 con 0x556557ace860 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557a259f0 con 0x556557acdf10 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31df240b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557a1ee80 con 0x556557acf460 2023-07-14T18:24:53.493 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48702/0 (socket says 172.21.15.64:48702) 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 -- 172.21.15.64:0/1531935707 learned_addr learned my addr 172.21.15.64:0/1531935707 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31d1645700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557acf8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acdf10 0x556557ace320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 -- 172.21.15.64:0/1531935707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 msgr2=0x556557acf8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557acf8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.494 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 -- 172.21.15.64:0/1531935707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acdf10 msgr2=0x556557ace320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acdf10 0x556557ace320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 -- 172.21.15.64:0/1531935707 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655784cd90 con 0x556557ace860 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31d1645700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557acf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31cbfff700 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005cd0 tx=0x7f31b8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fbe70ab00b6716b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.433+0000 7f31b6ffd700 1 -- 172.21.15.64:0/1531935707 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b800b6b0 con 0x556557ace860 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31b6ffd700 1 -- 172.21.15.64:0/1531935707 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31b801e040 con 0x556557ace860 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31b6ffd700 1 -- 172.21.15.64:0/1531935707 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b8004a30 con 0x556557ace860 2023-07-14T18:24:53.495 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 -- 172.21.15.64:0/1531935707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 msgr2=0x556557acee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f31b8005cd0 tx=0x7f31b8020000 comp rx=0 tx=0).stop 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 -- 172.21.15.64:0/1531935707 shutdown_connections 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557acf8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acdf10 0x556557ace320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 --2- 172.21.15.64:0/1531935707 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557ace860 0x556557acee80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 -- 172.21.15.64:0/1531935707 >> 172.21.15.64:0/1531935707 conn(0x556557ac3630 msgr2=0x556557ac8980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.496 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 -- 172.21.15.64:0/1531935707 shutdown_connections 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.434+0000 7f31df240b80 1 -- 172.21.15.64:0/1531935707 wait complete. 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 -- start start 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 Processor -- start 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 0x556557ad7e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 0x556557adeea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565579bea70 con 0x556557acdf10 2023-07-14T18:24:53.497 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565579be770 con 0x556557acf460 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565578a28d0 con 0x556557ace860 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31df240b80 10 client.0 _ll_register_callbacks cb 0x5565579bc710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48706/0 (socket says 172.21.15.64:48706) 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 -- 172.21.15.64:0/291480517 learned_addr learned my addr 172.21.15.64:0/291480517 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31cbfff700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 0x556557ad7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.498 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d1645700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 0x556557adeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 -- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 msgr2=0x556557ad7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 0x556557ad7e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 -- 172.21.15.64:0/291480517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 msgr2=0x556557adeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.435+0000 7f31d0e44700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 0x556557adeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31d0e44700 1 -- 172.21.15.64:0/291480517 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31cc1078b0 con 0x556557acdf10 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31d1645700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 0x556557adeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31cbfff700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 0x556557ad7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.499 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31d0e44700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f31c000efd0 tx=0x7f31c009b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c894d86aa5c86c85 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c00ce130 con 0x556557acdf10 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31c000bc70 con 0x556557acdf10 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565579784a0 con 0x556557acdf10 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31c009b460 con 0x556557acdf10 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 10 client.15459 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x556557835c10 con 0x556557acdf10 2023-07-14T18:24:53.500 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31c00b7020 con 0x556557acdf10 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f31c00a3420 con 0x556557acdf10 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 10 client.15459 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 10 client.15459 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 20 client.15459 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557ae2bf0 0x556557ae2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 4 client.15459 mds_command: new command op to 24487 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:53.501 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.436+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 --> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556557a9cd50 con 0x556557ae2bf0 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.437+0000 7f31cbfff700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557ae2bf0 0x556557ae2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.437+0000 7f31abfff700 10 client.15459 ms_handle_connect on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.437+0000 7f31cbfff700 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557ae2bf0 0x556557ae2050 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b57d5ac99bb42219 server_cookie=eda8606f5c08b4a5 in_seq=0 out_seq=0 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31abfff700 1 -- 172.21.15.64:0/291480517 <== mds.3 v2:172.21.15.144:6836/1582348642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x556557a9cd50 con 0x556557ae2bf0 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31abfff700 10 client.15459 handle_command_reply: tid=0 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 client.15459 shutdown 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 msgr2=0x556557ad7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f31c000efd0 tx=0x7f31c009b9a0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 shutdown_connections 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557ae2bf0 0x556557ae2050 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557acf460 0x556557adeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ad7920 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31abfff700 0 client.15459 ms_handle_reset on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 --2- 172.21.15.64:0/291480517 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557ace860 0x556557ad7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 >> 172.21.15.64:0/291480517 conn(0x556557ac3630 msgr2=0x556557ac7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 shutdown_connections 2023-07-14T18:24:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 1 -- 172.21.15.64:0/291480517 wait complete. 2023-07-14T18:24:53.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.438+0000 7f31df240b80 20 client.15459 trim_cache size 0 max 16384 2023-07-14T18:24:53.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.456+0000 7f31df240b80 1 Processor -- start 2023-07-14T18:24:53.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.456+0000 7f31df240b80 1 -- start start 2023-07-14T18:24:53.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.456+0000 7f31df240b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.456+0000 7f31df240b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557aeb940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.456+0000 7f31df240b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557ad2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31df240b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565579c8470 con 0x556557acdf10 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31df240b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565579ed790 con 0x556557ace860 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31df240b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565578aede0 con 0x556557acf460 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48714/0 (socket says 172.21.15.64:48714) 2023-07-14T18:24:53.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 -- 172.21.15.64:0/3494557258 learned_addr learned my addr 172.21.15.64:0/3494557258 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31cbfff700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557aeb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d1645700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557ad2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 -- 172.21.15.64:0/3494557258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 msgr2=0x556557ad2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557ad2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 -- 172.21.15.64:0/3494557258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 msgr2=0x556557aeb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557aeb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 -- 172.21.15.64:0/3494557258 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556557873200 con 0x556557acdf10 2023-07-14T18:24:53.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31cbfff700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557aeb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31d0e44700 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f31c009b3f0 tx=0x7f31c009ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8331579a2c51186 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31b77fe700 1 -- 172.21.15.64:0/3494557258 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c00ce130 con 0x556557acdf10 2023-07-14T18:24:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31b77fe700 1 -- 172.21.15.64:0/3494557258 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31c00b7030 con 0x556557acdf10 2023-07-14T18:24:53.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.457+0000 7f31b77fe700 1 -- 172.21.15.64:0/3494557258 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c00a38b0 con 0x556557acdf10 2023-07-14T18:24:53.515 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- 172.21.15.64:0/3494557258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 msgr2=0x556557ae2050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.515 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f31c009b3f0 tx=0x7f31c009ae70 comp rx=0 tx=0).stop 2023-07-14T18:24:53.515 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- 172.21.15.64:0/3494557258 shutdown_connections 2023-07-14T18:24:53.515 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acf460 0x556557ad2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.515 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acdf10 0x556557ae2050 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 --2- 172.21.15.64:0/3494557258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557aeb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- 172.21.15.64:0/3494557258 >> 172.21.15.64:0/3494557258 conn(0x556557ac3630 msgr2=0x556557ad5830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.516 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- 172.21.15.64:0/3494557258 shutdown_connections 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- 172.21.15.64:0/3494557258 wait complete. 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 -- start start 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.458+0000 7f31df240b80 1 Processor -- start 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557ae7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557aeab40 con 0x556557acf460 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557aeacb0 con 0x556557ace860 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x556557aeae70 con 0x556557acdf10 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 10 client.0 _ll_register_callbacks cb 0x5565579bc710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d0e44700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31cbfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557ae7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48724/0 (socket says 172.21.15.64:48724) 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 -- 172.21.15.64:0/1177730099 learned_addr learned my addr 172.21.15.64:0/1177730099 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:56376/0 (socket says 172.21.15.64:56376) 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 -- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 msgr2=0x556557ae7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 -- 172.21.15.64:0/1177730099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 msgr2=0x556557ae7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557ae7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 -- 172.21.15.64:0/1177730099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556557835c10 con 0x556557acf460 2023-07-14T18:24:53.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31d1645700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f31c40079c0 tx=0x7f31c4007cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0ce09d92abe13a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31abfff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c4007e60 con 0x556557acf460 2023-07-14T18:24:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f31c4013960 con 0x556557acf460 2023-07-14T18:24:53.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.459+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f31b801e040 con 0x556557acf460 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31c4013b20 con 0x556557acf460 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 10 client.15465 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55655784b260 con 0x556557acf460 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31b801e040 con 0x556557acf460 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f31c40163e0 con 0x556557acf460 2023-07-14T18:24:53.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 10 client.15465 resolve_mds: resolved name 'c' to daemon mds.c 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 20 client.15465 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557b3c650 0x556557b38fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 4 client.15465 mds_command: new command op to 24487 tid=0[{"prefix": "status"}] 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 --> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556557a17e60 con 0x556557b3c650 2023-07-14T18:24:53.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.460+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557b3c650 0x556557b38fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31abfff700 10 client.15465 ms_handle_connect on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31d0e44700 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557b3c650 0x556557b38fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bc87b1d46f3bf380 server_cookie=dbbeaf2618c2c686 in_seq=0 out_seq=0 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31abfff700 1 -- 172.21.15.64:0/1177730099 <== mds.3 v2:172.21.15.144:6836/1582348642 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x556557a17e60 con 0x556557b3c650 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31abfff700 10 client.15465 handle_command_reply: tid=0 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 client.15465 shutdown 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 msgr2=0x556557aea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.523 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f31c40079c0 tx=0x7f31c4007cd0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 shutdown_connections 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:6836/1582348642,v1:172.21.15.144:6837/1582348642] conn(0x556557b3c650 0x556557b38fc0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x556557acdf10 0x556557ae7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31abfff700 0 client.15465 ms_handle_reset on v2:172.21.15.144:6836/1582348642 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556557acf460 0x556557aea600 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 --2- 172.21.15.64:0/1177730099 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x556557ace860 0x556557ae7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 >> 172.21.15.64:0/1177730099 conn(0x556557ac3630 msgr2=0x556557ae2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 shutdown_connections 2023-07-14T18:24:53.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 1 -- 172.21.15.64:0/1177730099 wait complete. 2023-07-14T18:24:53.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.461+0000 7f31df240b80 20 client.15465 trim_cache size 0 max 16384 2023-07-14T18:24:53.525 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 3, 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24487, 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 87.207523108999993, 2023-07-14T18:24:53.526 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:53.527 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 125.35747957300001 2023-07-14T18:24:53.527 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:53.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 msgr2=0x7f31bc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 0x7f31bc0a3680 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f31b800b2e0 tx=0x7f31b8004000 comp rx=0 tx=0).stop 2023-07-14T18:24:53.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 msgr2=0x7f31cc1947a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f31c000ea80 tx=0x7f31c000ed90 comp rx=0 tx=0).stop 2023-07-14T18:24:53.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 shutdown_connections 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f31cc1006c0 0x7f31cc19bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f31bc0a11d0 0x7f31bc0a3680 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31cc0feb40 0x7f31cc1947a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 --2- 172.21.15.64:0/1910617737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f31cc0ffd30 0x7f31cc194ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 >> 172.21.15.64:0/1910617737 conn(0x7f31cc0fa160 msgr2=0x7f31cc0fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 shutdown_connections 2023-07-14T18:24:53.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.462+0000 7f31d30a8700 1 -- 172.21.15.64:0/1910617737 wait complete. 2023-07-14T18:24:53.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:53 smithi064 ceph-mon[96720]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-07-14T18:24:53.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:53 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/899996034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:53.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:53 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/900441260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.530 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:53 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/899694032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.531 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T18:24:53.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 -- 172.21.15.64:0/2839606751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 msgr2=0x7f67a8100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.660 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 --2- 172.21.15.64:0/2839606751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a8100150 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6790009990 tx=0x7f6790009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 -- 172.21.15.64:0/2839606751 shutdown_connections 2023-07-14T18:24:53.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 --2- 172.21.15.64:0/2839606751 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a8100690 0x7f67a8106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 --2- 172.21.15.64:0/2839606751 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a80feb10 0x7f67a80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.661 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 --2- 172.21.15.64:0/2839606751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a8100150 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 -- 172.21.15.64:0/2839606751 >> 172.21.15.64:0/2839606751 conn(0x7f67a80fa130 msgr2=0x7f67a80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.606+0000 7f67aeba1700 1 -- 172.21.15.64:0/2839606751 shutdown_connections 2023-07-14T18:24:53.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 -- 172.21.15.64:0/2839606751 wait complete. 2023-07-14T18:24:53.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 Processor -- start 2023-07-14T18:24:53.662 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 -- start start 2023-07-14T18:24:53.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a80feb10 0x7f67a8068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a8100690 0x7f67a819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a819f630 con 0x7f67a80ffd00 2023-07-14T18:24:53.663 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a819f7a0 con 0x7f67a8100690 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67aeba1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a819f960 con 0x7f67a80feb10 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48736/0 (socket says 172.21.15.64:48736) 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67a7fff700 1 -- 172.21.15.64:0/3570119964 learned_addr learned my addr 172.21.15.64:0/3570119964 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.607+0000 7f67ac93d700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a80feb10 0x7f67a8068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.664 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67ad13e700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a8100690 0x7f67a819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 -- 172.21.15.64:0/3570119964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a80feb10 msgr2=0x7f67a8068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a80feb10 0x7f67a8068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 -- 172.21.15.64:0/3570119964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a8100690 msgr2=0x7f67a819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a8100690 0x7f67a819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 -- 172.21.15.64:0/3570119964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67900095f0 con 0x7f67a80ffd00 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67900098e0 tx=0x7f6790004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a5ffb700 1 -- 172.21.15.64:0/3570119964 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679001d070 con 0x7f67a80ffd00 2023-07-14T18:24:53.665 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a5ffb700 1 -- 172.21.15.64:0/3570119964 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f679000bdd0 con 0x7f67a80ffd00 2023-07-14T18:24:53.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a819fbe0 con 0x7f67a80ffd00 2023-07-14T18:24:53.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67a5ffb700 1 -- 172.21.15.64:0/3570119964 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6790022d50 con 0x7f67a80ffd00 2023-07-14T18:24:53.666 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.608+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a81a0060 con 0x7f67a80ffd00 2023-07-14T18:24:53.676 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.610+0000 7f67a5ffb700 1 -- 172.21.15.64:0/3570119964 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f679000b940 con 0x7f67a80ffd00 2023-07-14T18:24:53.676 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.611+0000 7f67a5ffb700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 0x7f67940a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.611+0000 7f67a5ffb700 1 -- 172.21.15.64:0/3570119964 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f67900ce1b0 con 0x7f67a80ffd00 2023-07-14T18:24:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.611+0000 7f67ac93d700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 0x7f67940a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67ac93d700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 0x7f67940a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f67a80ffb60 tx=0x7f679800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 Processor -- start 2023-07-14T18:24:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 -- start start 2023-07-14T18:24:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7019320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae701a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f1dc60 con 0x558ae7019860 2023-07-14T18:24:53.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f709f0 con 0x558ae701a460 2023-07-14T18:24:53.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67bad39b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f69e80 con 0x558ae7018f10 2023-07-14T18:24:53.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48744/0 (socket says 172.21.15.64:48744) 2023-07-14T18:24:53.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.612+0000 7f67a7fff700 1 -- 172.21.15.64:0/2991380614 learned_addr learned my addr 172.21.15.64:0/2991380614 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67ac93d700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7019320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67ad13e700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae701a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 -- 172.21.15.64:0/2991380614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 msgr2=0x558ae7019320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7019320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 -- 172.21.15.64:0/2991380614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 msgr2=0x558ae701a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae701a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 -- 172.21.15.64:0/2991380614 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ae6d97d90 con 0x558ae7019860 2023-07-14T18:24:53.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67ac93d700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7019320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f67a7fff700 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f679009b040 tx=0x7f6790000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2c73cf8e200b6d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f679effd700 1 -- 172.21.15.64:0/2991380614 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679009f170 con 0x558ae7019860 2023-07-14T18:24:53.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f679effd700 1 -- 172.21.15.64:0/2991380614 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f679000bbf0 con 0x558ae7019860 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.613+0000 7f679effd700 1 -- 172.21.15.64:0/2991380614 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67900a10b0 con 0x558ae7019860 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- 172.21.15.64:0/2991380614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 msgr2=0x558ae7019e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f679009b040 tx=0x7f6790000bc0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- 172.21.15.64:0/2991380614 shutdown_connections 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7019320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae701a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 --2- 172.21.15.64:0/2991380614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7019e80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- 172.21.15.64:0/2991380614 >> 172.21.15.64:0/2991380614 conn(0x558ae700e630 msgr2=0x558ae7013980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- 172.21.15.64:0/2991380614 shutdown_connections 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- 172.21.15.64:0/2991380614 wait complete. 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 -- start start 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.614+0000 7f67bad39b80 1 Processor -- start 2023-07-14T18:24:53.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7022590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae7029b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f09a70 con 0x558ae7019860 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f09770 con 0x558ae701a460 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6ded8d0 con 0x558ae7018f10 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 10 client.0 _ll_register_callbacks cb 0x558ae6f07710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48750/0 (socket says 172.21.15.64:48750) 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 -- 172.21.15.64:0/821692442 learned_addr learned my addr 172.21.15.64:0/821692442 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67ac93d700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7022590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67ad13e700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae7029b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 -- 172.21.15.64:0/821692442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 msgr2=0x558ae7022590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7022590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 -- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 msgr2=0x558ae7029b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae7029b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 -- 172.21.15.64:0/821692442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a81a0060 con 0x558ae7019860 2023-07-14T18:24:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67ad13e700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae7029b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:53.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67ac93d700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7022590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67a7fff700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f67900cb9f0 tx=0x7f6790097680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745291ffe098c5c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f6787fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679009b340 con 0x558ae7019860 2023-07-14T18:24:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ae6ec34a0 con 0x558ae7019860 2023-07-14T18:24:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.615+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f679009a830 con 0x558ae7019860 2023-07-14T18:24:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f679009a9f0 con 0x558ae7019860 2023-07-14T18:24:53.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 10 client.15477 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558ae6d80c10 con 0x558ae7019860 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x558ae6ec34a0 con 0x558ae7019860 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f679009a2a0 con 0x558ae7019860 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 10 client.15477 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 20 client.15477 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae702d3a0 0x558ae702ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 4 client.15477 mds_command: new command op to 24500 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 --> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ae6fe7d50 con 0x558ae702d3a0 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.616+0000 7f67ac93d700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae702d3a0 0x558ae702ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.617+0000 7f67ac93d700 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae702d3a0 0x558ae702ca40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab18c1faa06dfe9a server_cookie=1fb71000581b43d4 in_seq=0 out_seq=0 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.617+0000 7f6787fff700 10 client.15477 ms_handle_connect on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.617+0000 7f6787fff700 1 -- 172.21.15.64:0/821692442 <== mds.2 v2:172.21.15.105:6836/2898950423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x558ae6fe7d50 con 0x558ae702d3a0 2023-07-14T18:24:53.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f6787fff700 10 client.15477 handle_command_reply: tid=0 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 client.15477 shutdown 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 msgr2=0x558ae7022ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f67900cb9f0 tx=0x7f6790097680 comp rx=0 tx=0).stop 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 shutdown_connections 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae702d3a0 0x558ae702ca40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae7018f10 0x558ae7022590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f6787fff700 0 client.15477 ms_handle_reset on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:53.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae701a460 0x558ae7029b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 --2- 172.21.15.64:0/821692442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7022ad0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 >> 172.21.15.64:0/821692442 conn(0x558ae700e630 msgr2=0x558ae7012600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 shutdown_connections 2023-07-14T18:24:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 1 -- 172.21.15.64:0/821692442 wait complete. 2023-07-14T18:24:53.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.618+0000 7f67bad39b80 20 client.15477 trim_cache size 0 max 16384 2023-07-14T18:24:53.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:53 smithi144 ceph-mon[98417]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-07-14T18:24:53.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:53 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/899996034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:53.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:53 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/900441260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.694 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:53 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/899694032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.635+0000 7f67bad39b80 1 Processor -- start 2023-07-14T18:24:53.698 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 -- start start 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae702ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7020500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6f709f0 con 0x558ae7019860 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae702e660 con 0x558ae7018f10 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67bad39b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae6fe7d50 con 0x558ae701a460 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.699 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48758/0 (socket says 172.21.15.64:48758) 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 -- 172.21.15.64:0/3397718046 learned_addr learned my addr 172.21.15.64:0/3397718046 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67ac93d700 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae702ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67ad13e700 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7020500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 -- 172.21.15.64:0/3397718046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 msgr2=0x558ae7020500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7020500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.700 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 -- 172.21.15.64:0/3397718046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 msgr2=0x558ae702ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae702ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 -- 172.21.15.64:0/3397718046 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ae6dbe200 con 0x558ae7019860 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f67a7fff700 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6790005cb0 tx=0x7f67900998f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d96853073aa8462a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f679f7fe700 1 -- 172.21.15.64:0/3397718046 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679009c170 con 0x558ae7019860 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f679f7fe700 1 -- 172.21.15.64:0/3397718046 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f679009aca0 con 0x558ae7019860 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.636+0000 7f679f7fe700 1 -- 172.21.15.64:0/3397718046 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679009c360 con 0x558ae7019860 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- 172.21.15.64:0/3397718046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 msgr2=0x558ae7029ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.701 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6790005cb0 tx=0x7f67900998f0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- 172.21.15.64:0/3397718046 shutdown_connections 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7020500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae702ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 --2- 172.21.15.64:0/3397718046 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7029ad0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- 172.21.15.64:0/3397718046 >> 172.21.15.64:0/3397718046 conn(0x558ae700e630 msgr2=0x558ae700fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- 172.21.15.64:0/3397718046 shutdown_connections 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- 172.21.15.64:0/3397718046 wait complete. 2023-07-14T18:24:53.702 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 -- start start 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.637+0000 7f67bad39b80 1 Processor -- start 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae7024ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7080fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae7081520 con 0x558ae7019860 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae7081660 con 0x558ae7018f10 2023-07-14T18:24:53.703 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ae70817a0 con 0x558ae701a460 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67bad39b80 10 client.0 _ll_register_callbacks cb 0x7f679009b110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48766/0 (socket says 172.21.15.64:48766) 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 -- 172.21.15.64:0/1892760185 learned_addr learned my addr 172.21.15.64:0/1892760185 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67ad13e700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7080fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67ac93d700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae7024ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 -- 172.21.15.64:0/1892760185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 msgr2=0x558ae7080fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7080fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 -- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 msgr2=0x558ae7024ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae7024ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 -- 172.21.15.64:0/1892760185 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558ae6d80c10 con 0x558ae7019860 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f67a7fff700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f679009b2d0 tx=0x7f6790000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01efd66c4a3b69f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f6787fff700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.638+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67900a3470 con 0x558ae7019860 2023-07-14T18:24:53.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f679000bbf0 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f679009b340 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f679009f170 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 10 client.15483 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558ae6d96260 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f679000bbf0 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f679009a830 con 0x558ae7019860 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.706 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 10 client.15483 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 20 client.15483 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae7085df0 0x558ae70829a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 4 client.15483 mds_command: new command op to 24500 tid=0[{"prefix": "status"}] 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 --> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558ae6f62e60 con 0x558ae7085df0 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.639+0000 7f67ac93d700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae7085df0 0x558ae70829a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f6787fff700 10 client.15483 ms_handle_connect on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:53.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67ac93d700 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae7085df0 0x558ae70829a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7f7dbd82328720b server_cookie=99b0b661da6c8fd in_seq=0 out_seq=0 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f6787fff700 1 -- 172.21.15.64:0/1892760185 <== mds.2 v2:172.21.15.105:6836/2898950423 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x558ae6f62e60 con 0x558ae7085df0 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f6787fff700 10 client.15483 handle_command_reply: tid=0 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 client.15483 shutdown 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 msgr2=0x558ae7025240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f679009b2d0 tx=0x7f6790000bc0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 shutdown_connections 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:6836/2898950423,v1:172.21.15.105:6837/2898950423] conn(0x558ae7085df0 0x558ae70829a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x558ae701a460 0x558ae7080fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x558ae7018f10 0x558ae7024ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f6787fff700 0 client.15483 ms_handle_reset on v2:172.21.15.105:6836/2898950423 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 --2- 172.21.15.64:0/1892760185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ae7019860 0x558ae7025240 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 >> 172.21.15.64:0/1892760185 conn(0x558ae700e630 msgr2=0x558ae702ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 shutdown_connections 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.640+0000 7f67bad39b80 1 -- 172.21.15.64:0/1892760185 wait complete. 2023-07-14T18:24:53.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67bad39b80 20 client.15483 trim_cache size 0 max 16384 2023-07-14T18:24:53.710 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:53.710 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:53.710 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 2, 2023-07-14T18:24:53.710 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24500, 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 88.398365228000003, 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:53.711 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 123.357420539 2023-07-14T18:24:53.712 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 msgr2=0x7f67940a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 0x7f67940a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f67a80ffb60 tx=0x7f679800b040 comp rx=0 tx=0).stop 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 msgr2=0x7f67a819ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67900098e0 tx=0x7f6790004730 comp rx=0 tx=0).stop 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 shutdown_connections 2023-07-14T18:24:53.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f67a80feb10 0x7f67a8068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f67940a11d0 0x7f67940a3680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67a8100690 0x7f67a819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 --2- 172.21.15.64:0/3570119964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67a80ffd00 0x7f67a819ca30 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 >> 172.21.15.64:0/3570119964 conn(0x7f67a80fa130 msgr2=0x7f67a80fc4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 shutdown_connections 2023-07-14T18:24:53.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.641+0000 7f67aeba1700 1 -- 172.21.15.64:0/3570119964 wait complete. 2023-07-14T18:24:53.715 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T18:24:53.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:53 smithi105 ceph-mon[98013]: pgmap v310: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2023-07-14T18:24:53.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:53 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/899996034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:24:53.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:53 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/900441260' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:53 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/899694032' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:24:53.841 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 -- 172.21.15.64:0/154572925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e78100690 msgr2=0x7f4e78106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.841 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/154572925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e78100690 0x7f4e78106f50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4e7000b0a0 tx=0x7f4e7000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 -- 172.21.15.64:0/154572925 shutdown_connections 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/154572925 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e78100690 0x7f4e78106f50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/154572925 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e780feb10 0x7f4e780fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/154572925 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e78100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 -- 172.21.15.64:0/154572925 >> 172.21.15.64:0/154572925 conn(0x7f4e780fa130 msgr2=0x7f4e780fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 -- 172.21.15.64:0/154572925 shutdown_connections 2023-07-14T18:24:53.842 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.789+0000 7f4e7da7c700 1 -- 172.21.15.64:0/154572925 wait complete. 2023-07-14T18:24:53.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 Processor -- start 2023-07-14T18:24:53.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 -- start start 2023-07-14T18:24:53.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e7819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.844 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 0x7f4e7819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7819d560 con 0x7f4e780feb10 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7819d6d0 con 0x7f4e78100690 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e7da7c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7819d890 con 0x7f4e780ffd00 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e777fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e777fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48776/0 (socket says 172.21.15.64:48776) 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e777fe700 1 -- 172.21.15.64:0/2188773522 learned_addr learned my addr 172.21.15.64:0/2188773522 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.845 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e7819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.790+0000 7f4e77fff700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 0x7f4e7819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 -- 172.21.15.64:0/2188773522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 msgr2=0x7f4e7819a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e7819a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 -- 172.21.15.64:0/2188773522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 msgr2=0x7f4e7819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 0x7f4e7819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 -- 172.21.15.64:0/2188773522 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009d00 con 0x7f4e780feb10 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e77fff700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 0x7f4e7819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.846 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e7819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4e68007e60 tx=0x7f4e6800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e74ff9700 1 -- 172.21.15.64:0/2188773522 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800de40 con 0x7f4e780feb10 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e78194570 con 0x7f4e780feb10 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e74ff9700 1 -- 172.21.15.64:0/2188773522 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e6801a070 con 0x7f4e780feb10 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e74ff9700 1 -- 172.21.15.64:0/2188773522 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68021b20 con 0x7f4e780feb10 2023-07-14T18:24:53.847 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.791+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e78194a80 con 0x7f4e780feb10 2023-07-14T18:24:53.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.793+0000 7f4e74ff9700 1 -- 172.21.15.64:0/2188773522 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4e68021c80 con 0x7f4e780feb10 2023-07-14T18:24:53.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.794+0000 7f4e74ff9700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 0x7f4e600a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.794+0000 7f4e74ff9700 1 -- 172.21.15.64:0/2188773522 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e6801c070 con 0x7f4e780feb10 2023-07-14T18:24:53.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.794+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 0x7f4e600a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.852 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.794+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 0x7f4e600a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4e780680a0 tx=0x7f4e6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.794+0000 7f4e89c14b80 1 Processor -- start 2023-07-14T18:24:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 -- start start 2023-07-14T18:24:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f461860 0x55669f461e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.853 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 0x55669f4628d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f365c60 con 0x55669f460f10 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f3b89f0 con 0x55669f461860 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f3b1e80 con 0x55669f462460 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48784/0 (socket says 172.21.15.64:48784) 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 -- 172.21.15.64:0/3022988789 learned_addr learned my addr 172.21.15.64:0/3022988789 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.854 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f461860 0x55669f461e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e77fff700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 0x55669f4628d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 -- 172.21.15.64:0/3022988789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 msgr2=0x55669f4628d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 0x55669f4628d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 -- 172.21.15.64:0/3022988789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f461860 msgr2=0x55669f461e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f461860 0x55669f461e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 -- 172.21.15.64:0/3022988789 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55669f1dfd90 con 0x55669f460f10 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e77fff700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 0x55669f4628d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e777fe700 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4e680aa130 tx=0x7f4e680a1d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5f568868d84041 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.795+0000 7f4e5dffb700 1 -- 172.21.15.64:0/3022988789 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a2210 con 0x55669f460f10 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e5dffb700 1 -- 172.21.15.64:0/3022988789 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e680a6240 con 0x55669f460f10 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e5dffb700 1 -- 172.21.15.64:0/3022988789 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a1990 con 0x55669f460f10 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3022988789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 msgr2=0x55669f461320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4e680aa130 tx=0x7f4e680a1d50 comp rx=0 tx=0).stop 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3022988789 shutdown_connections 2023-07-14T18:24:53.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f462460 0x55669f4628d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f460f10 0x55669f461320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3022988789 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f461860 0x55669f461e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3022988789 >> 172.21.15.64:0/3022988789 conn(0x55669f456630 msgr2=0x55669f45b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3022988789 shutdown_connections 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.796+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3022988789 wait complete. 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 -- start start 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 Processor -- start 2023-07-14T18:24:53.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f460f10 0x55669f46a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f461860 0x55669f46aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f351a70 con 0x55669f462460 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f351770 con 0x55669f460f10 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f2358d0 con 0x55669f461860 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e89c14b80 10 client.0 _ll_register_callbacks cb 0x55669f34f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.858 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48794/0 (socket says 172.21.15.64:48794) 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 -- 172.21.15.64:0/2392550756 learned_addr learned my addr 172.21.15.64:0/2392550756 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f460f10 0x55669f46a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 -- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f461860 msgr2=0x55669f46aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f461860 0x55669f46aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f461860 0x55669f46aef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 -- 172.21.15.64:0/2392550756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f460f10 msgr2=0x55669f46a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f460f10 0x55669f46a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.797+0000 7f4e77fff700 1 -- 172.21.15.64:0/2392550756 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e78194a80 con 0x55669f462460 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e77fff700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4e70015000 tx=0x7f4e70007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850ba162e8f461fc server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7000e040 con 0x55669f462460 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e70004a30 con 0x55669f462460 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55669f30b4a0 con 0x55669f462460 2023-07-14T18:24:53.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e70004bf0 con 0x55669f462460 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 10 client.15495 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55669f1c8c10 con 0x55669f462460 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55669f30b4a0 con 0x55669f462460 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e70024030 con 0x55669f462460 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 10 client.15495 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 20 client.15495 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f475c80 0x55669f475030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 4 client.15495 mds_command: new command op to 24508 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.798+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 --> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55669f42fd50 con 0x55669f475c80 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.799+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f475c80 0x55669f475030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.799+0000 7f4e56ffd700 10 client.15495 ms_handle_connect on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.799+0000 7f4e777fe700 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f475c80 0x55669f475030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=72e0ffabd7e0a9cd server_cookie=4f30ffd677160f55 in_seq=0 out_seq=0 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e56ffd700 1 -- 172.21.15.64:0/2392550756 <== mds.1 v2:172.21.15.144:6838/4051776603 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55669f42fd50 con 0x55669f475c80 2023-07-14T18:24:53.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e56ffd700 10 client.15495 handle_command_reply: tid=0 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 client.15495 shutdown 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 msgr2=0x55669f471f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4e70015000 tx=0x7f4e70007dd0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 shutdown_connections 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f475c80 0x55669f475030 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f462460 0x55669f471f30 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e56ffd700 0 client.15495 ms_handle_reset on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:53.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f460f10 0x55669f46a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/2392550756 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f461860 0x55669f46aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 >> 172.21.15.64:0/2392550756 conn(0x55669f456630 msgr2=0x55669f45a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 shutdown_connections 2023-07-14T18:24:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 1 -- 172.21.15.64:0/2392550756 wait complete. 2023-07-14T18:24:53.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.800+0000 7f4e89c14b80 20 client.15495 trim_cache size 0 max 16384 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 Processor -- start 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 -- start start 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f45b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f472760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f42fd50 con 0x55669f461860 2023-07-14T18:24:53.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f30b4a0 con 0x55669f462460 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f46bba0 con 0x55669f460f10 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48800/0 (socket says 172.21.15.64:48800) 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 -- 172.21.15.64:0/3411351569 learned_addr learned my addr 172.21.15.64:0/3411351569 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e777fe700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f45b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e77fff700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f472760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.872 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 -- 172.21.15.64:0/3411351569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 msgr2=0x55669f45b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f45b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 -- 172.21.15.64:0/3411351569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 msgr2=0x55669f472760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f472760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e76ffd700 1 -- 172.21.15.64:0/3411351569 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55669f206200 con 0x55669f461860 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e77fff700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f472760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.818+0000 7f4e777fe700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f45b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:53.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c010fa0 tx=0x7f4e6c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c126ef44c1b48dea server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e5e7fc700 1 -- 172.21.15.64:0/3411351569 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c011e60 con 0x55669f461860 2023-07-14T18:24:53.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e5e7fc700 1 -- 172.21.15.64:0/3411351569 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e6c015070 con 0x55669f461860 2023-07-14T18:24:53.874 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e5e7fc700 1 -- 172.21.15.64:0/3411351569 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c020730 con 0x55669f461860 2023-07-14T18:24:53.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3411351569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 msgr2=0x55669f47e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c010fa0 tx=0x7f4e6c013000 comp rx=0 tx=0).stop 2023-07-14T18:24:53.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3411351569 shutdown_connections 2023-07-14T18:24:53.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f45b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f472760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/3411351569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47e9f0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3411351569 >> 172.21.15.64:0/3411351569 conn(0x55669f456630 msgr2=0x55669f469620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3411351569 shutdown_connections 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.819+0000 7f4e89c14b80 1 -- 172.21.15.64:0/3411351569 wait complete. 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 -- start start 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 Processor -- start 2023-07-14T18:24:53.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f47bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f47d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f47d770 con 0x55669f461860 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f47d8e0 con 0x55669f462460 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55669f47daa0 con 0x55669f460f10 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e89c14b80 10 client.0 _ll_register_callbacks cb 0x7f4e6c005ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:53.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e76ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e76ffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48806/0 (socket says 172.21.15.64:48806) 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e76ffd700 1 -- 172.21.15.64:0/14984659 learned_addr learned my addr 172.21.15.64:0/14984659 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e777fe700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f47bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.820+0000 7f4e76ffd700 1 -- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 msgr2=0x55669f47bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e77fff700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f47d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f47bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e76ffd700 1 -- 172.21.15.64:0/14984659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 msgr2=0x55669f47d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:53.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f47d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e76ffd700 1 -- 172.21.15.64:0/14984659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6c010c50 con 0x55669f461860 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e76ffd700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c011010 tx=0x7f4e6c029720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3364b687672fb8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c02d050 con 0x55669f461860 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4e6c017070 con 0x55669f461860 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4e680a6240 con 0x55669f461860 2023-07-14T18:24:53.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4e6c029d00 con 0x55669f461860 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 10 client.15501 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55669f1de260 con 0x55669f461860 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4e680a6240 con 0x55669f461860 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4e6c0204b0 con 0x55669f461860 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 10 client.15501 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 10 client.15501 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.821+0000 7f4e89c14b80 20 client.15501 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:53.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f4cf120 0x55669f4cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e89c14b80 4 client.15501 mds_command: new command op to 24508 tid=0[{"prefix": "status"}] 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 --> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55669f3aae60 con 0x55669f4cf120 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e777fe700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f4cf120 0x55669f4cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e56ffd700 10 client.15501 ms_handle_connect on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e777fe700 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f4cf120 0x55669f4cbab0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=48500cebddc18f58 server_cookie=5c0837624eb8cef9 in_seq=0 out_seq=0 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e56ffd700 1 -- 172.21.15.64:0/14984659 <== mds.1 v2:172.21.15.144:6838/4051776603 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55669f3aae60 con 0x55669f4cf120 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e56ffd700 10 client.15501 handle_command_reply: tid=0 2023-07-14T18:24:53.883 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.822+0000 7f4e89c14b80 1 client.15501 shutdown 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 msgr2=0x55669f47c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c011010 tx=0x7f4e6c029720 comp rx=0 tx=0).stop 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 shutdown_connections 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:6838/4051776603,v1:172.21.15.144:6839/4051776603] conn(0x55669f4cf120 0x55669f4cbab0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55669f460f10 0x55669f47bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55669f462460 0x55669f47d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.884 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 --2- 172.21.15.64:0/14984659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55669f461860 0x55669f47c220 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 >> 172.21.15.64:0/14984659 conn(0x55669f456630 msgr2=0x55669f472760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e56ffd700 0 client.15501 ms_handle_reset on v2:172.21.15.144:6838/4051776603 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 shutdown_connections 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 1 -- 172.21.15.64:0/14984659 wait complete. 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e89c14b80 20 client.15501 trim_cache size 0 max 16384 2023-07-14T18:24:53.885 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 1, 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24508, 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:53.886 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 89.590398829999998, 2023-07-14T18:24:53.887 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:53.887 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:53.887 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:53.887 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 121.226020842 2023-07-14T18:24:53.887 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 msgr2=0x7f4e600a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.888 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 0x7f4e600a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f4e780680a0 tx=0x7f4e6c009260 comp rx=0 tx=0).stop 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 msgr2=0x7f4e78068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4e68007e60 tx=0x7f4e6800dc60 comp rx=0 tx=0).stop 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 shutdown_connections 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4e600a1150 0x7f4e600a3600 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4e78100690 0x7f4e7819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4e780feb10 0x7f4e78068330 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.889 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 --2- 172.21.15.64:0/2188773522 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4e780ffd00 0x7f4e7819a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 >> 172.21.15.64:0/2188773522 conn(0x7f4e780fa130 msgr2=0x7f4e78103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 shutdown_connections 2023-07-14T18:24:53.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.823+0000 7f4e7da7c700 1 -- 172.21.15.64:0/2188773522 wait complete. 2023-07-14T18:24:53.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T18:24:54.018 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 -- 172.21.15.64:0/2105319443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 msgr2=0x7fe1101053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.018 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 --2- 172.21.15.64:0/2105319443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe1101053f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe104009990 tx=0x7fe104009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.018 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 -- 172.21.15.64:0/2105319443 shutdown_connections 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 --2- 172.21.15.64:0/2105319443 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 0x7fe110107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 --2- 172.21.15.64:0/2105319443 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 0x7fe110102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 --2- 172.21.15.64:0/2105319443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe1101053f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 -- 172.21.15.64:0/2105319443 >> 172.21.15.64:0/2105319443 conn(0x7fe1100fa130 msgr2=0x7fe1100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 -- 172.21.15.64:0/2105319443 shutdown_connections 2023-07-14T18:24:54.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.966+0000 7fe115bed700 1 -- 172.21.15.64:0/2105319443 wait complete. 2023-07-14T18:24:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 Processor -- start 2023-07-14T18:24:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 -- start start 2023-07-14T18:24:54.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 0x7fe110195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 0x7fe11019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11019d3d0 con 0x7fe110103010 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11019d510 con 0x7fe1101006f0 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe115bed700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11019d6a0 con 0x7fe110105930 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48820/0 (socket says 172.21.15.64:48820) 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.967+0000 7fe10effd700 1 -- 172.21.15.64:0/3259560121 learned_addr learned my addr 172.21.15.64:0/3259560121 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 0x7fe110195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10ffff700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 0x7fe11019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 -- 172.21.15.64:0/3259560121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 msgr2=0x7fe11019ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 0x7fe11019ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 -- 172.21.15.64:0/3259560121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 msgr2=0x7fe110195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.023 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 0x7fe110195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 -- 172.21.15.64:0/3259560121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1040095f0 con 0x7fe110103010 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10effd700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe104006010 tx=0x7fe104004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10cff9700 1 -- 172.21.15.64:0/3259560121 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10401d070 con 0x7fe110103010 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10cff9700 1 -- 172.21.15.64:0/3259560121 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe1040041b0 con 0x7fe110103010 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe11019d920 con 0x7fe110103010 2023-07-14T18:24:54.024 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe10cff9700 1 -- 172.21.15.64:0/3259560121 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104022da0 con 0x7fe110103010 2023-07-14T18:24:54.025 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.968+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11019dd40 con 0x7fe110103010 2023-07-14T18:24:54.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.970+0000 7fe10cff9700 1 -- 172.21.15.64:0/3259560121 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe104022570 con 0x7fe110103010 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.971+0000 7fe10cff9700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 0x7fe0f80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.971+0000 7fe10cff9700 1 -- 172.21.15.64:0/3259560121 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe1040ce200 con 0x7fe110103010 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.971+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 0x7fe0f80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 0x7fe0f80a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe100005fd0 tx=0x7fe10000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 Processor -- start 2023-07-14T18:24:54.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 -- start start 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1533320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc15348d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1437c60 con 0x55efc1533860 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc148a9f0 con 0x55efc1532f10 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe121d85b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1483e80 con 0x55efc1534460 2023-07-14T18:24:54.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48828/0 (socket says 172.21.15.64:48828) 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 -- 172.21.15.64:0/1245018646 learned_addr learned my addr 172.21.15.64:0/1245018646 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1533320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 -- 172.21.15.64:0/1245018646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 msgr2=0x55efc15348d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10ffff700 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc15348d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc15348d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 -- 172.21.15.64:0/1245018646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 msgr2=0x55efc1533320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1533320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.972+0000 7fe10effd700 1 -- 172.21.15.64:0/1245018646 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efc12b1d90 con 0x55efc1533860 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe10effd700 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe104097830 tx=0x7fe104009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aadda679d0450813 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe0f5ffb700 1 -- 172.21.15.64:0/1245018646 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10409b3b0 con 0x55efc1533860 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe0f5ffb700 1 -- 172.21.15.64:0/1245018646 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe10409f360 con 0x55efc1533860 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe0f5ffb700 1 -- 172.21.15.64:0/1245018646 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10409aa80 con 0x55efc1533860 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 -- 172.21.15.64:0/1245018646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 msgr2=0x55efc1533e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe104097830 tx=0x7fe104009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 -- 172.21.15.64:0/1245018646 shutdown_connections 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc15348d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1533320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1245018646 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1533e80 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.973+0000 7fe121d85b80 1 -- 172.21.15.64:0/1245018646 >> 172.21.15.64:0/1245018646 conn(0x55efc1528630 msgr2=0x55efc152d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.974+0000 7fe121d85b80 1 -- 172.21.15.64:0/1245018646 shutdown_connections 2023-07-14T18:24:54.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.974+0000 7fe121d85b80 1 -- 172.21.15.64:0/1245018646 wait complete. 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.974+0000 7fe121d85b80 1 -- start start 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.974+0000 7fe121d85b80 1 Processor -- start 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc153c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc1543b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1423a70 con 0x55efc1533860 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1423770 con 0x55efc1532f10 2023-07-14T18:24:54.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc13078d0 con 0x55efc1534460 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 10 client.0 _ll_register_callbacks cb 0x55efc1421710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48834/0 (socket says 172.21.15.64:48834) 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 -- 172.21.15.64:0/3422231662 learned_addr learned my addr 172.21.15.64:0/3422231662 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc153c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10ffff700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc1543b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 -- 172.21.15.64:0/3422231662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 msgr2=0x55efc1543b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc1543b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 -- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 msgr2=0x55efc153c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc153c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 -- 172.21.15.64:0/3422231662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe11019dd40 con 0x55efc1533860 2023-07-14T18:24:54.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe10effd700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe10409afe0 tx=0x7fe1040977a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b93e7a0887c73 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe0eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1040a36f0 con 0x55efc1533860 2023-07-14T18:24:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe1040a3260 con 0x55efc1533860 2023-07-14T18:24:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.975+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55efc13dd4a0 con 0x55efc1533860 2023-07-14T18:24:54.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1040a3420 con 0x55efc1533860 2023-07-14T18:24:54.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 10 client.15513 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:54.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55efc129ac10 con 0x55efc1533860 2023-07-14T18:24:54.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe1040c9020 con 0x55efc1533860 2023-07-14T18:24:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe104097240 con 0x55efc1533860 2023-07-14T18:24:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 10 client.15513 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 10 client.15513 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:24:54.044 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 20 client.15513 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc1547900 0x55efc1546ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 4 client.15513 mds_command: new command op to 24545 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T18:24:54.045 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efc1501d50 con 0x55efc1547900 2023-07-14T18:24:54.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.976+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc1547900 0x55efc1546ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe0eeffd700 10 client.15513 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:54.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc1547900 0x55efc1546ab0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56a6345018d4827 server_cookie=48a15bc09099583e in_seq=0 out_seq=0 2023-07-14T18:24:54.046 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3422231662 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55efc1501d50 con 0x55efc1547900 2023-07-14T18:24:54.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe0eeffd700 10 client.15513 handle_command_reply: tid=0 2023-07-14T18:24:54.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe121d85b80 1 client.15513 shutdown 2023-07-14T18:24:54.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.977+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 msgr2=0x55efc153cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.047 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fe10409afe0 tx=0x7fe1040977a0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 shutdown_connections 2023-07-14T18:24:54.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc1547900 0x55efc1546ab0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1534460 0x55efc1543b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.048 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc153c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe0eeffd700 0 client.15513 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3422231662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc153cb40 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 >> 172.21.15.64:0/3422231662 conn(0x55efc1528630 msgr2=0x55efc152c600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 shutdown_connections 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 1 -- 172.21.15.64:0/3422231662 wait complete. 2023-07-14T18:24:54.049 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.978+0000 7fe121d85b80 20 client.15513 trim_cache size 0 max 16384 2023-07-14T18:24:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.995+0000 7fe121d85b80 1 Processor -- start 2023-07-14T18:24:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.995+0000 7fe121d85b80 1 -- start start 2023-07-14T18:24:54.054 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1532f10 0x55efc153a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 0x55efc1537320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1534d50 con 0x55efc1533860 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc148a9f0 con 0x55efc1534460 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe121d85b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc152f470 con 0x55efc1532f10 2023-07-14T18:24:54.055 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48842/0 (socket says 172.21.15.64:48842) 2023-07-14T18:24:54.056 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 -- 172.21.15.64:0/1769356133 learned_addr learned my addr 172.21.15.64:0/1769356133 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:54.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1532f10 0x55efc153a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10ffff700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 0x55efc1537320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.057 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 -- 172.21.15.64:0/1769356133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1532f10 msgr2=0x55efc153a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1532f10 0x55efc153a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 -- 172.21.15.64:0/1769356133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 msgr2=0x55efc1537320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 0x55efc1537320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.058 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 -- 172.21.15.64:0/1769356133 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efc12d8200 con 0x55efc1533860 2023-07-14T18:24:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10ffff700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 0x55efc1537320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe10effd700 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe10409b030 tx=0x7fe1040998e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8727d2c2f25f0b8c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.059 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe0f67fc700 1 -- 172.21.15.64:0/1769356133 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1040c5c20 con 0x55efc1533860 2023-07-14T18:24:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe0f67fc700 1 -- 172.21.15.64:0/1769356133 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55efc13dd4a0 con 0x55efc1533860 2023-07-14T18:24:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.996+0000 7fe0f67fc700 1 -- 172.21.15.64:0/1769356133 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe104084b80 con 0x55efc1533860 2023-07-14T18:24:54.060 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- 172.21.15.64:0/1769356133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 msgr2=0x55efc1550060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe10409b030 tx=0x7fe1040998e0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- 172.21.15.64:0/1769356133 shutdown_connections 2023-07-14T18:24:54.061 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1532f10 0x55efc153a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1534460 0x55efc1537320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 --2- 172.21.15.64:0/1769356133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1533860 0x55efc1550060 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- 172.21.15.64:0/1769356133 >> 172.21.15.64:0/1769356133 conn(0x55efc1528630 msgr2=0x55efc153b260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.062 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- 172.21.15.64:0/1769356133 shutdown_connections 2023-07-14T18:24:54.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- 172.21.15.64:0/1769356133 wait complete. 2023-07-14T18:24:54.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 -- start start 2023-07-14T18:24:54.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.997+0000 7fe121d85b80 1 Processor -- start 2023-07-14T18:24:54.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1539100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 0x55efc1539640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc1539d80 con 0x55efc1534460 2023-07-14T18:24:54.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc159b4b0 con 0x55efc1532f10 2023-07-14T18:24:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efc159b620 con 0x55efc1533860 2023-07-14T18:24:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe121d85b80 10 client.0 _ll_register_callbacks cb 0x7fe10409b1d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T18:24:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10effd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 0x55efc1539640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48850/0 (socket says 172.21.15.64:48850) 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 -- 172.21.15.64:0/3048548747 learned_addr learned my addr 172.21.15.64:0/3048548747 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1539100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 -- 172.21.15.64:0/3048548747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 msgr2=0x55efc1539640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 0x55efc1539640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 -- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 msgr2=0x55efc1539100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T18:24:54.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1539100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 -- 172.21.15.64:0/3048548747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efc129ac10 con 0x55efc1534460 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1539100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10effd700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 0x55efc1539640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe10ffff700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe1080078f0 tx=0x7fe108007c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49753d5fba4cee0d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.998+0000 7fe0eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe10800e070 con 0x55efc1534460 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mon.0 v2:172.21.15.64:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe108018650 con 0x55efc1534460 2023-07-14T18:24:54.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe10409f360 con 0x55efc1534460 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe108018810 con 0x55efc1534460 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 10 client.15519 fetch_fsmap learned FSMap version 27 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55efc12b0260 con 0x55efc1534460 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mon.0 v2:172.21.15.64:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe10409f360 con 0x55efc1534460 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mon.0 v2:172.21.15.64:3300/0 5 ==== fsmap(e 27) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fe108016aa0 con 0x55efc1534460 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2023-07-14T18:24:54.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 10 client.15519 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 20 client.15519 populate_metadata read hostname 'smithi064' 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc15a0370 0x55efc159cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 4 client.15519 mds_command: new command op to 24545 tid=0[{"prefix": "status"}] 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 --> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55efc147ce60 con 0x55efc15a0370 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:53.999+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc15a0370 0x55efc159cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe10f7fe700 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc15a0370 0x55efc159cd00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2774d414304dbf8 server_cookie=7958337235d43b50 in_seq=0 out_seq=0 2023-07-14T18:24:54.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe0eeffd700 10 client.15519 ms_handle_connect on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe0eeffd700 1 -- 172.21.15.64:0/3048548747 <== mds.0 v2:172.21.15.105:6838/2402376208 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55efc147ce60 con 0x55efc15a0370 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe0eeffd700 10 client.15519 handle_command_reply: tid=0 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 client.15519 shutdown 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 msgr2=0x55efc159af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe1080078f0 tx=0x7fe108007c00 comp rx=0 tx=0).stop 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 shutdown_connections 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:6838/2402376208,v1:172.21.15.105:6839/2402376208] conn(0x55efc15a0370 0x55efc159cd00 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55efc1534460 0x55efc159af70 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe0eeffd700 0 client.15519 ms_handle_reset on v2:172.21.15.105:6838/2402376208 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x55efc1532f10 0x55efc1539100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 --2- 172.21.15.64:0/3048548747 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x55efc1533860 0x55efc1539640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 >> 172.21.15.64:0/3048548747 conn(0x55efc1528630 msgr2=0x55efc153c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 shutdown_connections 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 1 -- 172.21.15.64:0/3048548747 wait complete. 2023-07-14T18:24:54.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.000+0000 7fe121d85b80 20 client.15519 trim_cache size 0 max 16384 2023-07-14T18:24:54.072 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "cluster_fsid": "5441c8fe-2272-11ee-9b33-001a4aab830c", 2023-07-14T18:24:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "whoami": 0, 2023-07-14T18:24:54.072 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 24545, 2023-07-14T18:24:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "want_state": "up:active", 2023-07-14T18:24:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "state": "up:active", 2023-07-14T18:24:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs", 2023-07-14T18:24:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "rank_uptime": 108.584311895, 2023-07-14T18:24:54.073 INFO:teuthology.orchestra.run.smithi064.stdout: "mdsmap_epoch": 27, 2023-07-14T18:24:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch": 87, 2023-07-14T18:24:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T18:24:54.074 INFO:teuthology.orchestra.run.smithi064.stdout: "uptime": 114.68941542100001 2023-07-14T18:24:54.074 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 msgr2=0x7fe0f80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 0x7fe0f80a3600 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe100005fd0 tx=0x7fe10000a560 comp rx=0 tx=0).stop 2023-07-14T18:24:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 msgr2=0x7fe110195d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:54.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe104006010 tx=0x7fe104004730 comp rx=0 tx=0).stop 2023-07-14T18:24:54.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 shutdown_connections 2023-07-14T18:24:54.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fe110105930 0x7fe11019ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fe0f80a1150 0x7fe0f80a3600 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe1101006f0 0x7fe110195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 --2- 172.21.15.64:0/3259560121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe110103010 0x7fe110195d90 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 >> 172.21.15.64:0/3259560121 conn(0x7fe1100fa130 msgr2=0x7fe110101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 shutdown_connections 2023-07-14T18:24:54.077 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:54.001+0000 7fe115bed700 1 -- 172.21.15.64:0/3259560121 wait complete. 2023-07-14T18:24:54.078 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-07-14T18:24:54.094 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-07-14T18:24:54.095 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-07-14T18:24:54.095 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2023-07-14T18:24:54.096 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T18:24:54.096 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:54.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:54.097 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2023-07-14T18:24:54.115 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-07-14T18:24:54.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:54.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T18:24:54.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:54.196 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2023-07-14T18:24:54.297 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-07-14T18:24:54.297 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T18:24:54.863 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:24:55.650 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:55 smithi064 ceph-mon[96720]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2023-07-14T18:24:55.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:55 smithi144 ceph-mon[98417]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2023-07-14T18:24:55.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:55 smithi105 ceph-mon[98013]: pgmap v311: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 0 B/s wr, 11 op/s 2023-07-14T18:24:56.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 -- 172.21.15.64:0/990911322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 msgr2=0x7f4c1810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:56.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 --2- 172.21.15.64:0/990911322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c1810ba60 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000b0d0 tx=0x7f4c1000b3e0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 -- 172.21.15.64:0/990911322 shutdown_connections 2023-07-14T18:24:56.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 --2- 172.21.15.64:0/990911322 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180fee40 0x7f4c180ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 --2- 172.21.15.64:0/990911322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c1810ba60 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 --2- 172.21.15.64:0/990911322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180ff7e0 0x7f4c181037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 -- 172.21.15.64:0/990911322 >> 172.21.15.64:0/990911322 conn(0x7f4c18075ca0 msgr2=0x7f4c180760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:56.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 -- 172.21.15.64:0/990911322 shutdown_connections 2023-07-14T18:24:56.065 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.011+0000 7f4c20314700 1 -- 172.21.15.64:0/990911322 wait complete. 2023-07-14T18:24:56.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 Processor -- start 2023-07-14T18:24:56.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 -- start start 2023-07-14T18:24:56.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 0x7f4c1819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:56.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 0x7f4c1819afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:56.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18195220 con 0x7f4c18103ee0 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18195390 con 0x7f4c180fee40 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c20314700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c18195550 con 0x7f4c180ff7e0 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c1e8b1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c1e8b1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48862/0 (socket says 172.21.15.64:48862) 2023-07-14T18:24:56.069 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.012+0000 7f4c1e8b1700 1 -- 172.21.15.64:0/2609849065 learned_addr learned my addr 172.21.15.64:0/2609849065 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:56.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1d8af700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 0x7f4c1819afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:56.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e0b0700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 0x7f4c1819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:56.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 -- 172.21.15.64:0/2609849065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 msgr2=0x7f4c1819afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:56.070 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 0x7f4c1819afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 -- 172.21.15.64:0/2609849065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 msgr2=0x7f4c1819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:56.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 0x7f4c1819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 -- 172.21.15.64:0/2609849065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c10009d00 con 0x7f4c18103ee0 2023-07-14T18:24:56.071 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1d8af700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 0x7f4c1819afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:56.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e0b0700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 0x7f4c1819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:56.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c1e8b1700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009380 tx=0x7f4c10009460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:56.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000e040 con 0x7f4c18103ee0 2023-07-14T18:24:56.072 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4c10003e10 con 0x7f4c18103ee0 2023-07-14T18:24:56.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c181957d0 con 0x7f4c18103ee0 2023-07-14T18:24:56.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c10012a80 con 0x7f4c18103ee0 2023-07-14T18:24:56.073 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.013+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c181a1a00 con 0x7f4c18103ee0 2023-07-14T18:24:56.074 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.014+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1804e340 con 0x7f4c18103ee0 2023-07-14T18:24:56.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.015+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4c10012be0 con 0x7f4c18103ee0 2023-07-14T18:24:56.075 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.016+0000 7f4c0f7fe700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 0x7f4c040a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:56.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.016+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4c100ce300 con 0x7f4c18103ee0 2023-07-14T18:24:56.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.016+0000 7f4c1e0b0700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 0x7f4c040a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:56.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.016+0000 7f4c1e0b0700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 0x7f4c040a3680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4c14005950 tx=0x7f4c140058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:56.076 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.020+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4c10097770 con 0x7f4c18103ee0 2023-07-14T18:24:56.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.264+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4c181a1d10 con 0x7f4c18103ee0 2023-07-14T18:24:56.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.265+0000 7f4c0f7fe700 1 -- 172.21.15.64:0/2609849065 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+88 (secure 0 0 0) 0x7f4c1009ad90 con 0x7f4c18103ee0 2023-07-14T18:24:56.319 INFO:teuthology.orchestra.run.smithi064.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T18:24:56.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.266+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 msgr2=0x7f4c040a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:56.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 0x7f4c040a3680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4c14005950 tx=0x7f4c140058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 msgr2=0x7f4c18194a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:56.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f4c10009380 tx=0x7f4c10009460 comp rx=0 tx=0).stop 2023-07-14T18:24:56.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 shutdown_connections 2023-07-14T18:24:56.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f4c040a11d0 0x7f4c040a3680 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4c18103ee0 0x7f4c18194a70 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4c180fee40 0x7f4c1819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 --2- 172.21.15.64:0/2609849065 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4c180ff7e0 0x7f4c1819afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:56.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 >> 172.21.15.64:0/2609849065 conn(0x7f4c18075ca0 msgr2=0x7f4c180fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:56.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 shutdown_connections 2023-07-14T18:24:56.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:56.267+0000 7f4c20314700 1 -- 172.21.15.64:0/2609849065 wait complete. 2023-07-14T18:24:56.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:56 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2609849065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:24:56.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:56 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2609849065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:24:56.711 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T18:24:56.712 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T18:24:56.712 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi064.front.sepia.ceph.com 2023-07-14T18:24:56.712 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T18:24:56.712 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:56.713 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T18:24:56.713 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T18:24:56.713 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T18:24:56.713 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T18:24:56.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:56.714 DEBUG:teuthology.orchestra.run.smithi064:> ip addr 2023-07-14T18:24:56.733 INFO:teuthology.orchestra.run.smithi064.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T18:24:56.733 INFO:teuthology.orchestra.run.smithi064.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T18:24:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T18:24:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:24:56.734 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 ::1/128 scope host 2023-07-14T18:24:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:24:56.735 INFO:teuthology.orchestra.run.smithi064.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:24:56.735 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:78 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:24:56.735 INFO:teuthology.orchestra.run.smithi064.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:24:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:a9:a3:79 brd ff:ff:ff:ff:ff:ff 2023-07-14T18:24:56.736 INFO:teuthology.orchestra.run.smithi064.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T18:24:56.736 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:de brd ff:ff:ff:ff:ff:ff 2023-07-14T18:24:56.737 INFO:teuthology.orchestra.run.smithi064.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T18:24:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: link/ether 0c:c4:7a:b6:f8:df brd ff:ff:ff:ff:ff:ff 2023-07-14T18:24:56.737 INFO:teuthology.orchestra.run.smithi064.stdout: inet 172.21.15.64/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T18:24:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft 41569sec preferred_lft 41569sec 2023-07-14T18:24:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: inet6 fe80::bf40:81a3:b7e0:f244/64 scope link noprefixroute 2023-07-14T18:24:56.738 INFO:teuthology.orchestra.run.smithi064.stdout: valid_lft forever preferred_lft forever 2023-07-14T18:24:56.738 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T18:24:56.739 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:56.739 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:24:56.740 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:24:56.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add name ceph-brx type bridge 2023-07-14T18:24:56.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr flush dev ceph-brx 2023-07-14T18:24:56.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set ceph-brx up 2023-07-14T18:24:56.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T18:24:56.742 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:24:56.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:56 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2609849065' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T18:24:56.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:56.857 DEBUG:teuthology.orchestra.run.smithi064:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T18:24:56.886 INFO:teuthology.orchestra.run.smithi064.stdout:1 2023-07-14T18:24:56.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:56.888 DEBUG:teuthology.orchestra.run.smithi064:> route 2023-07-14T18:24:56.947 INFO:teuthology.orchestra.run.smithi064.stdout:Kernel IP routing table 2023-07-14T18:24:56.947 INFO:teuthology.orchestra.run.smithi064.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T18:24:56.947 INFO:teuthology.orchestra.run.smithi064.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T18:24:56.947 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T18:24:56.948 INFO:teuthology.orchestra.run.smithi064.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T18:24:56.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:56.948 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:24:56.949 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:24:56.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T18:24:56.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T18:24:56.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T18:24:56.949 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:24:57.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.056 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2023-07-14T18:24:57.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.074 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list-id 2023-07-14T18:24:57.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.132 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:24:57.132 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:24:57.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T18:24:57.133 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T18:24:57.133 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:24:57.227 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T18:24:57.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.228 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:24:57.228 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:24:57.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T18:24:57.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T18:24:57.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T18:24:57.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T18:24:57.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T18:24:57.229 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:24:57.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi064:> (cd / && exec bash -c ' 2023-07-14T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi064:> set -e 2023-07-14T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set brx.0 up 2023-07-14T18:24:57.434 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T18:24:57.435 DEBUG:teuthology.orchestra.run.smithi064:> ') 2023-07-14T18:24:57.535 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:57 smithi064 ceph-mon[96720]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2023-07-14T18:24:57.544 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2023-07-14T18:24:57.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:24:57.545 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:57.569 INFO:teuthology.orchestra.run.smithi064.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T18:24:57.569 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T18:24:57.570 DEBUG:teuthology.orchestra.run.smithi064:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:57.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-07-14T18:24:57.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:57 smithi144 ceph-mon[98417]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2023-07-14T18:24:57.695 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:24:57.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:24:57.763 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:24:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:24:57.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.765 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-07-14T18:24:57.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:57 smithi105 ceph-mon[98013]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2023-07-14T18:24:57.821 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T18:24:57.822 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/sv_0/4b2cb415-bdae-42ca-a661-4eaba754ca09 --client_fs=cephfs) 2023-07-14T18:24:57.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo modprobe fuse 2023-07-14T18:24:57.891 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T18:24:57.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T18:24:57.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi064.stderr:2023-07-14T18:24:57.902+0000 7fdf6898f5c0 -1 init, newargv = 0x55abbd0a1040 newargc=15 2023-07-14T18:24:57.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi064.stderr:ceph-fuse[147999]: starting ceph client 2023-07-14T18:24:57.962 INFO:teuthology.orchestra.run.smithi064.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T18:24:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T18:24:57.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:57.965 DEBUG:teuthology.orchestra.run.smithi064:> ls /sys/fs/fuse/connections 2023-07-14T18:24:57.978 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi064.stderr:ceph-fuse[147999]: starting fuse 2023-07-14T18:24:57.995 INFO:teuthology.orchestra.run.smithi064.stdout:124 2023-07-14T18:24:57.995 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T18:24:57.995 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T18:24:57.996 DEBUG:teuthology.orchestra.run.smithi064:> import glob 2023-07-14T18:24:57.996 DEBUG:teuthology.orchestra.run.smithi064:> import re 2023-07-14T18:24:57.996 DEBUG:teuthology.orchestra.run.smithi064:> import os 2023-07-14T18:24:57.996 DEBUG:teuthology.orchestra.run.smithi064:> import subprocess 2023-07-14T18:24:57.997 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:24:57.997 DEBUG:teuthology.orchestra.run.smithi064:> def _find_admin_socket(client_name): 2023-07-14T18:24:57.997 DEBUG:teuthology.orchestra.run.smithi064:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-07-14T18:24:57.997 DEBUG:teuthology.orchestra.run.smithi064:> files = glob.glob(asok_path) 2023-07-14T18:24:57.997 DEBUG:teuthology.orchestra.run.smithi064:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-07-14T18:24:57.998 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:24:57.998 DEBUG:teuthology.orchestra.run.smithi064:> # Given a non-glob path, it better be there 2023-07-14T18:24:57.998 DEBUG:teuthology.orchestra.run.smithi064:> if "*" not in asok_path: 2023-07-14T18:24:57.998 DEBUG:teuthology.orchestra.run.smithi064:> assert(len(files) == 1) 2023-07-14T18:24:57.999 DEBUG:teuthology.orchestra.run.smithi064:> return files[0] 2023-07-14T18:24:57.999 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:24:57.999 DEBUG:teuthology.orchestra.run.smithi064:> for f in files: 2023-07-14T18:24:57.999 DEBUG:teuthology.orchestra.run.smithi064:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T18:24:57.999 DEBUG:teuthology.orchestra.run.smithi064:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T18:24:58.000 DEBUG:teuthology.orchestra.run.smithi064:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T18:24:58.000 DEBUG:teuthology.orchestra.run.smithi064:> contents = proc_f.read() 2023-07-14T18:24:58.000 DEBUG:teuthology.orchestra.run.smithi064:> if mountpoint in contents: 2023-07-14T18:24:58.000 DEBUG:teuthology.orchestra.run.smithi064:> return f 2023-07-14T18:24:58.001 DEBUG:teuthology.orchestra.run.smithi064:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T18:24:58.001 DEBUG:teuthology.orchestra.run.smithi064:> 2023-07-14T18:24:58.001 DEBUG:teuthology.orchestra.run.smithi064:> print(_find_admin_socket("client.0")) 2023-07-14T18:24:58.001 DEBUG:teuthology.orchestra.run.smithi064:> ' 2023-07-14T18:24:58.120 INFO:teuthology.orchestra.run.smithi064.stdout:/var/run/ceph/ceph-client.0.147999.asok 2023-07-14T18:24:58.302 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.147999.asok 2023-07-14T18:24:58.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:58.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.147999.asok status 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout: "metadata": { 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout: "entity_id": "0", 2023-07-14T18:24:58.464 INFO:teuthology.orchestra.run.smithi064.stdout: "hostname": "smithi064", 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "pid": "147999", 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "root": "/volumes/_nogroup/sv_0/4b2cb415-bdae-42ca-a661-4eaba754ca09" 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_count": 0, 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "dentry_pinned_count": 0, 2023-07-14T18:24:58.465 INFO:teuthology.orchestra.run.smithi064.stdout: "id": 15531, 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "inst": { 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "name": { 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "client", 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "num": 15531 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-07-14T18:24:58.466 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3158985028 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": { 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type": "v1", 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: "addr": "192.168.0.1:0", 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: "nonce": 3158985028 2023-07-14T18:24:58.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "inst_str": "client.15531 192.168.0.1:0/3158985028", 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "addr_str": "192.168.0.1:0/3158985028", 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "inode_count": 5, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_epoch": 27, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch": 87, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_epoch_barrier": 0, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "blocklisted": false, 2023-07-14T18:24:58.468 INFO:teuthology.orchestra.run.smithi064.stdout: "fs_name": "cephfs" 2023-07-14T18:24:58.469 INFO:teuthology.orchestra.run.smithi064.stdout:} 2023-07-14T18:24:58.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:58.474 DEBUG:teuthology.orchestra.run.smithi064:> stat --file-system '--printf=%T 2023-07-14T18:24:58.474 DEBUG:teuthology.orchestra.run.smithi064:> ' -- /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:58.493 INFO:teuthology.orchestra.run.smithi064.stdout:fuseblk 2023-07-14T18:24:58.493 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:58.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:24:58.494 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T18:24:58.564 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T18:24:58.575 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T18:24:58.575 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:24:58.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T18:24:58.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 -- 172.21.15.64:0/966841978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 msgr2=0x7f29080fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:58.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 --2- 172.21.15.64:0/966841978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f29080fef40 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2904009990 tx=0x7f2904009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 -- 172.21.15.64:0/966841978 shutdown_connections 2023-07-14T18:24:58.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 --2- 172.21.15.64:0/966841978 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2908100770 0x7f2908107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 --2- 172.21.15.64:0/966841978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f29080fef40 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 --2- 172.21.15.64:0/966841978 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f29080ffdc0 0x7f2908100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 -- 172.21.15.64:0/966841978 >> 172.21.15.64:0/966841978 conn(0x7f29080fa130 msgr2=0x7f29080fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:58.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 -- 172.21.15.64:0/966841978 shutdown_connections 2023-07-14T18:24:58.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.677+0000 7f290f3db700 1 -- 172.21.15.64:0/966841978 wait complete. 2023-07-14T18:24:58.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.678+0000 7f290f3db700 1 Processor -- start 2023-07-14T18:24:58.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.678+0000 7f290f3db700 1 -- start start 2023-07-14T18:24:58.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:58.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 0x7f2908194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2908100770 0x7f2908195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2908196230 con 0x7f29080feb30 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29081963a0 con 0x7f29080ffdc0 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2908196560 con 0x7f2908100770 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48894/0 (socket says 172.21.15.64:48894) 2023-07-14T18:24:58.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 -- 172.21.15.64:0/3525456752 learned_addr learned my addr 172.21.15.64:0/3525456752 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290c976700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 0x7f2908194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d978700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2908100770 0x7f2908195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 -- 172.21.15.64:0/3525456752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2908100770 msgr2=0x7f2908195bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2908100770 0x7f2908195bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 -- 172.21.15.64:0/3525456752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 msgr2=0x7f2908194d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:58.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 0x7f2908194d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 -- 172.21.15.64:0/3525456752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29040095f0 con 0x7f29080feb30 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290c976700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 0x7f2908194d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290d177700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2904006010 tx=0x7f2904004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290401d070 con 0x7f29080feb30 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29081967e0 con 0x7f29080feb30 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f29040041b0 con 0x7f29080feb30 2023-07-14T18:24:58.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2904022da0 con 0x7f29080feb30 2023-07-14T18:24:58.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.679+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290819d000 con 0x7f29080feb30 2023-07-14T18:24:58.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.680+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29080fde30 con 0x7f29080feb30 2023-07-14T18:24:58.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.681+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2904022570 con 0x7f29080feb30 2023-07-14T18:24:58.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.682+0000 7f28fe7fc700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 0x7f28f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:58.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.682+0000 7f290c976700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 0x7f28f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:58.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.682+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f29040ce5c0 con 0x7f29080feb30 2023-07-14T18:24:58.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.683+0000 7f290c976700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 0x7f28f40a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f28f8005950 tx=0x7f28f80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:58.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.685+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2904097a30 con 0x7f29080feb30 2023-07-14T18:24:58.983 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:58.930+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f2908105b40 con 0x7f29080feb30 2023-07-14T18:24:59.416 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.363+0000 7f28fe7fc700 1 -- 172.21.15.64:0/3525456752 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f290409b050 con 0x7f29080feb30 2023-07-14T18:24:59.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 msgr2=0x7f28f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:59.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 0x7f28f40a3600 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f28f8005950 tx=0x7f28f80058e0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 msgr2=0x7f2908194810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:59.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2904006010 tx=0x7f2904004730 comp rx=0 tx=0).stop 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 shutdown_connections 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f2908100770 0x7f2908195bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f28f40a1150 0x7f28f40a3600 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29080feb30 0x7f2908194810 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 --2- 172.21.15.64:0/3525456752 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29080ffdc0 0x7f2908194d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.420 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 >> 172.21.15.64:0/3525456752 conn(0x7f29080fa130 msgr2=0x7f29080fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:59.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 shutdown_connections 2023-07-14T18:24:59.421 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.366+0000 7f290f3db700 1 -- 172.21.15.64:0/3525456752 wait complete. 2023-07-14T18:24:59.433 DEBUG:teuthology.orchestra.run.smithi064:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01' 2023-07-14T18:24:59.568 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 -- 172.21.15.64:0/3145791529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 msgr2=0x7f8aa0100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 --2- 172.21.15.64:0/3145791529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa0100230 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009990 tx=0x7f8a8c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 -- 172.21.15.64:0/3145791529 shutdown_connections 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 --2- 172.21.15.64:0/3145791529 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa00feb30 0x7f8aa00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 --2- 172.21.15.64:0/3145791529 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa0100770 0x7f8aa0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.513+0000 7f8aa712c700 1 --2- 172.21.15.64:0/3145791529 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa0100230 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- 172.21.15.64:0/3145791529 >> 172.21.15.64:0/3145791529 conn(0x7f8aa00fa130 msgr2=0x7f8aa00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:24:59.569 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- 172.21.15.64:0/3145791529 shutdown_connections 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- 172.21.15.64:0/3145791529 wait complete. 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 Processor -- start 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- start start 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa00feb30 0x7f8aa01011a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa01016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:59.570 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0100770 0x7f8aa0101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa019f8d0 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa019fa10 con 0x7f8aa00feb30 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.514+0000 7f8aa712c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa019fb50 con 0x7f8aa0100770 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa01016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa01016e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48910/0 (socket says 172.21.15.64:48910) 2023-07-14T18:24:59.571 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 -- 172.21.15.64:0/3305436812 learned_addr learned my addr 172.21.15.64:0/3305436812 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8aa56c9700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0100770 0x7f8aa0101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8aa4ec8700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa00feb30 0x7f8aa01011a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 -- 172.21.15.64:0/3305436812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0100770 msgr2=0x7f8aa0101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f8aa0100770 0x7f8aa0101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 -- 172.21.15.64:0/3305436812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa00feb30 msgr2=0x7f8aa01011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:24:59.572 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa00feb30 0x7f8aa01011a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 -- 172.21.15.64:0/3305436812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c0095f0 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8aa4ec8700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8aa00feb30 0x7f8aa01011a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9ffff700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8aa00ffdc0 0x7f8aa01016e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c000c00 tx=0x7f8a8c003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c01d070 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8aa712c700 1 -- 172.21.15.64:0/3305436812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa019fd80 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a8c00bd70 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c022d20 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.515+0000 7f8aa712c700 1 -- 172.21.15.64:0/3305436812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa01a01b0 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.574 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.517+0000 7f8aa712c700 1 -- 172.21.15.64:0/3305436812 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aa004e420 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.518+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8a8c022470 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.518+0000 7f8a9dffb700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a900a1150 0x7f8a900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:24:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.519+0000 7f8aa4ec8700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a900a1150 0x7f8a900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:24:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.519+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a8c0ce930 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.519+0000 7f8aa4ec8700 1 --2- 172.21.15.64:0/3305436812 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f8a900a1150 0x7f8a900a3600 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f8a94005fd0 tx=0x7f8a94005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:24:59.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.522+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a8c097e00 con 0x7f8aa00ffdc0 2023-07-14T18:24:59.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:59 smithi144 ceph-mon[98417]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2023-07-14T18:24:59.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:24:59 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:24:59.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.758+0000 7f8aa712c700 1 -- 172.21.15.64:0/3305436812 --> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}) v1 -- 0x7f8aa0068160 con 0x7f8a900a1150 2023-07-14T18:24:59.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:59 smithi105 ceph-mon[98013]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2023-07-14T18:24:59.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:24:59 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:24:59.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-mon[96720]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2023-07-14T18:24:59.818 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T18:24:59.819 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.763+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:24:59.819 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.763+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:24:59.819 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.763+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:24:59.819 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.764+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:24:59.819 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.764+0000 7f94c2e84700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T18:24:59.830 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:24:59.778+0000 7f8a9dffb700 1 -- 172.21.15.64:0/3305436812 <== mgr.14152 v2:172.21.15.64:6800/387190897 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8aa0068160 con 0x7f8a900a1150 2023-07-14T18:24:59.832 INFO:teuthology.orchestra.run.smithi064.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T18:24:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:24:59.836 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T18:24:59.948 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=59b0efa8feea4d35a002c94bc4564d2a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi064 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup random 0.01'" 2023-07-14T18:24:59.950 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-07-14T18:24:59.960 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-07-14T18:24:59.961 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:24:59.961 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:24:59.985 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:24:59.985 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:25:00.043 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:25:00.092 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:24:59 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:24:59.777+0000 7f94c2e84700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T18:25:00.209 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 -- 172.21.15.64:0/3133898324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15701017c0 msgr2=0x7f1570108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 --2- 172.21.15.64:0/3133898324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15701017c0 0x7f1570108080 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f156800b0a0 tx=0x7f156800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 -- 172.21.15.64:0/3133898324 shutdown_connections 2023-07-14T18:25:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 --2- 172.21.15.64:0/3133898324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15701017c0 0x7f1570108080 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 --2- 172.21.15.64:0/3133898324 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 0x7f1570100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 --2- 172.21.15.64:0/3133898324 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1570100e30 0x7f1570101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 -- 172.21.15.64:0/3133898324 >> 172.21.15.64:0/3133898324 conn(0x7f15700fb260 msgr2=0x7f15700fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:00.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 -- 172.21.15.64:0/3133898324 shutdown_connections 2023-07-14T18:25:00.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.156+0000 7f1574c64700 1 -- 172.21.15.64:0/3133898324 wait complete. 2023-07-14T18:25:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 Processor -- start 2023-07-14T18:25:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 -- start start 2023-07-14T18:25:00.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 0x7f15700743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15701017c0 0x7f1570072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570073710 con 0x7f1570100e30 2023-07-14T18:25:00.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1570073880 con 0x7f15700ffc40 2023-07-14T18:25:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f1574c64700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157019f8d0 con 0x7f15701017c0 2023-07-14T18:25:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48944/0 (socket says 172.21.15.64:48944) 2023-07-14T18:25:00.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 -- 172.21.15.64:0/593653412 learned_addr learned my addr 172.21.15.64:0/593653412 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156ed9d700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15701017c0 0x7f1570072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156e59c700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 0x7f15700743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 -- 172.21.15.64:0/593653412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15701017c0 msgr2=0x7f1570072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15701017c0 0x7f1570072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 -- 172.21.15.64:0/593653412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 msgr2=0x7f15700743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 0x7f15700743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.218 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.157+0000 7f156dd9b700 1 -- 172.21.15.64:0/593653412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009d00 con 0x7f1570100e30 2023-07-14T18:25:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f156dd9b700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f156000d950 tx=0x7f156000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156000f890 con 0x7f1570100e30 2023-07-14T18:25:00.219 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157019fbb0 con 0x7f1570100e30 2023-07-14T18:25:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f156000fe80 con 0x7f1570100e30 2023-07-14T18:25:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156000e3b0 con 0x7f1570100e30 2023-07-14T18:25:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.158+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15701a00c0 con 0x7f1570100e30 2023-07-14T18:25:00.220 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.159+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1570107080 con 0x7f1570100e30 2023-07-14T18:25:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.160+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1560010460 con 0x7f1570100e30 2023-07-14T18:25:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.160+0000 7f155f7fe700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 0x7f15580a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.161+0000 7f156e59c700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 0x7f15580a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.221 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.161+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f15600cc9a0 con 0x7f1570100e30 2023-07-14T18:25:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.161+0000 7f156e59c700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 0x7f15580a3650 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1570100c90 tx=0x7f156400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.222 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.164+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1560095de0 con 0x7f1570100e30 2023-07-14T18:25:00.417 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.364+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f157002c8e0 con 0x7f1570100e30 2023-07-14T18:25:00.419 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.366+0000 7f155f7fe700 1 -- 172.21.15.64:0/593653412 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7f1560020090 con 0x7f1570100e30 2023-07-14T18:25:00.420 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:25:00.420 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:25:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 msgr2=0x7f15580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 0x7f15580a3650 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1570100c90 tx=0x7f156400a3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 msgr2=0x7f1570072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.422 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f156000d950 tx=0x7f156000dc60 comp rx=0 tx=0).stop 2023-07-14T18:25:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 shutdown_connections 2023-07-14T18:25:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f15701017c0 0x7f1570072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f15580a11a0 0x7f15580a3650 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f15700ffc40 0x7f15700743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.423 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 --2- 172.21.15.64:0/593653412 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1570100e30 0x7f1570072a30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 >> 172.21.15.64:0/593653412 conn(0x7f15700fb260 msgr2=0x7f15701041c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 shutdown_connections 2023-07-14T18:25:00.424 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.369+0000 7f1574c64700 1 -- 172.21.15.64:0/593653412 wait complete. 2023-07-14T18:25:00.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:25:00.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:25:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/1092896980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f40feb10 msgr2=0x7ff4f40fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.611 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/1092896980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f40feb10 0x7ff4f40fef20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7ff4e400b0a0 tx=0x7ff4e400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/1092896980 shutdown_connections 2023-07-14T18:25:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/1092896980 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f4100690 0x7ff4f4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/1092896980 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f40feb10 0x7ff4f40fef20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.612 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/1092896980 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40ffd00 0x7ff4f4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/1092896980 >> 172.21.15.64:0/1092896980 conn(0x7ff4f40fa130 msgr2=0x7ff4f40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/1092896980 shutdown_connections 2023-07-14T18:25:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.556+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/1092896980 wait complete. 2023-07-14T18:25:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 Processor -- start 2023-07-14T18:25:00.613 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 -- start start 2023-07-14T18:25:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40feb10 0x7ff4f4194950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f40ffd00 0x7ff4f4194e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.614 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f419c4d0 con 0x7ff4f4100690 2023-07-14T18:25:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f419c610 con 0x7ff4f40feb10 2023-07-14T18:25:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4fb0cb700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4f419c7a0 con 0x7ff4f40ffd00 2023-07-14T18:25:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48958/0 (socket says 172.21.15.64:48958) 2023-07-14T18:25:00.615 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 -- 172.21.15.64:0/258566223 learned_addr learned my addr 172.21.15.64:0/258566223 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f8e67700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40feb10 0x7ff4f4194950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f3fff700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f40ffd00 0x7ff4f4194e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 -- 172.21.15.64:0/258566223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f40ffd00 msgr2=0x7ff4f4194e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f40ffd00 0x7ff4f4194e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.616 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 -- 172.21.15.64:0/258566223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40feb10 msgr2=0x7ff4f4194950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40feb10 0x7ff4f4194950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.557+0000 7ff4f9668700 1 -- 172.21.15.64:0/258566223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e4009d00 con 0x7ff4f4100690 2023-07-14T18:25:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4f9668700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00d980 tx=0x7ff4ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec004a30 con 0x7ff4f4100690 2023-07-14T18:25:00.617 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4ec0054b0 con 0x7ff4f4100690 2023-07-14T18:25:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4f419ca80 con 0x7ff4f4100690 2023-07-14T18:25:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ec00fab0 con 0x7ff4f4100690 2023-07-14T18:25:00.618 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.558+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4f419cf90 con 0x7ff4f4100690 2023-07-14T18:25:00.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.559+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4f41061a0 con 0x7ff4f4100690 2023-07-14T18:25:00.619 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.560+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff4ec00fc10 con 0x7ff4f4100690 2023-07-14T18:25:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.561+0000 7ff4f1ffb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 0x7ff4e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.561+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4ec0ccb50 con 0x7ff4f4100690 2023-07-14T18:25:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.561+0000 7ff4f8e67700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 0x7ff4e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.620 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.561+0000 7ff4f8e67700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 0x7ff4e00a3690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4007a30 tx=0x7ff4e400b7b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.621 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.565+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4ec095dd0 con 0x7ff4f4100690 2023-07-14T18:25:00.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:00 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:25:00.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:00 smithi144 ceph-mon[98417]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:25:00.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:00 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:25:00.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:00 smithi105 ceph-mon[98013]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:25:00.816 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.763+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4f402caf0 con 0x7ff4f4100690 2023-07-14T18:25:00.817 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.764+0000 7ff4f1ffb700 1 -- 172.21.15.64:0/258566223 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff4ec0170a0 con 0x7ff4f4100690 2023-07-14T18:25:00.818 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:25:00.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 msgr2=0x7ff4e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 0x7ff4e00a3690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4007a30 tx=0x7ff4e400b7b0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.819 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 msgr2=0x7ff4f419bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff4ec00d980 tx=0x7ff4ec00dc90 comp rx=0 tx=0).stop 2023-07-14T18:25:00.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 shutdown_connections 2023-07-14T18:25:00.820 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7ff4e00a11e0 0x7ff4e00a3690 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff4f4100690 0x7ff4f419bf00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff4f40feb10 0x7ff4f4194950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 --2- 172.21.15.64:0/258566223 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7ff4f40ffd00 0x7ff4f4194e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.821 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 >> 172.21.15.64:0/258566223 conn(0x7ff4f40fa130 msgr2=0x7ff4f41032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:00.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 shutdown_connections 2023-07-14T18:25:00.822 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.766+0000 7ff4fb0cb700 1 -- 172.21.15.64:0/258566223 wait complete. 2023-07-14T18:25:00.830 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:00 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3525456752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T18:25:00.831 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:00 smithi064 ceph-mon[96720]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2023-07-14T18:25:00.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:25:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/1778474283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740feb10 msgr2=0x7f6d740fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.962 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/1778474283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740feb10 0x7f6d740fef20 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f6d64009990 tx=0x7f6d64009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/1778474283 shutdown_connections 2023-07-14T18:25:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/1778474283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d74100690 0x7f6d74106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/1778474283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740feb10 0x7f6d740fef20 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.963 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/1778474283 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d740ffd00 0x7f6d74100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/1778474283 >> 172.21.15.64:0/1778474283 conn(0x7f6d740fa130 msgr2=0x7f6d740fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/1778474283 shutdown_connections 2023-07-14T18:25:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/1778474283 wait complete. 2023-07-14T18:25:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 Processor -- start 2023-07-14T18:25:00.964 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.909+0000 7f6d7a9cd700 1 -- start start 2023-07-14T18:25:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d740feb10 0x7f6d74068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.966 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d74100690 0x7f6d7419ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7419d480 con 0x7f6d740ffd00 2023-07-14T18:25:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7419d5f0 con 0x7f6d740feb10 2023-07-14T18:25:00.967 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d7a9cd700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7419d7b0 con 0x7f6d74100690 2023-07-14T18:25:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48970/0 (socket says 172.21.15.64:48970) 2023-07-14T18:25:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 -- 172.21.15.64:0/189330268 learned_addr learned my addr 172.21.15.64:0/189330268 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:00.968 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d73fff700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d740feb10 0x7f6d74068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d78f6a700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d74100690 0x7f6d7419ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 -- 172.21.15.64:0/189330268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d74100690 msgr2=0x7f6d7419ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d74100690 0x7f6d7419ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.969 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 -- 172.21.15.64:0/189330268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d740feb10 msgr2=0x7f6d74068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d740feb10 0x7f6d74068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 -- 172.21.15.64:0/189330268 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d640095f0 con 0x7f6d740ffd00 2023-07-14T18:25:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d737fe700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00d980 tx=0x7f6d5c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.910+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c004220 con 0x7f6d740ffd00 2023-07-14T18:25:00.970 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.911+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6d5c007640 con 0x7f6d740ffd00 2023-07-14T18:25:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.911+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7419da90 con 0x7f6d740ffd00 2023-07-14T18:25:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.911+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d5c0055e0 con 0x7f6d740ffd00 2023-07-14T18:25:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.911+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7419dfa0 con 0x7f6d740ffd00 2023-07-14T18:25:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.911+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d7410a1e0 con 0x7f6d740ffd00 2023-07-14T18:25:00.971 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.912+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6d5c004750 con 0x7f6d740ffd00 2023-07-14T18:25:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.913+0000 7f6d717fa700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 0x7f6d600a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.913+0000 7f6d73fff700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 0x7f6d600a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:00.972 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.913+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6d5c0ccc40 con 0x7f6d740ffd00 2023-07-14T18:25:00.973 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.914+0000 7f6d73fff700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 0x7f6d600a3690 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400c010 tx=0x7f6d64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:00.974 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:00.918+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6d5c0d3080 con 0x7f6d740ffd00 2023-07-14T18:25:01.167 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.114+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6d74105af0 con 0x7f6d740ffd00 2023-07-14T18:25:01.168 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.115+0000 7f6d717fa700 1 -- 172.21.15.64:0/189330268 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6d5c0160a0 con 0x7f6d740ffd00 2023-07-14T18:25:01.168 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:25:01.170 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 msgr2=0x7f6d600a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 0x7f6d600a3690 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f6d6400c010 tx=0x7f6d64005fb0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 msgr2=0x7f6d7419a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6d5c00d980 tx=0x7f6d5c00dc90 comp rx=0 tx=0).stop 2023-07-14T18:25:01.171 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 shutdown_connections 2023-07-14T18:25:01.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f6d74100690 0x7f6d7419ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f6d600a11e0 0x7f6d600a3690 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6d740feb10 0x7f6d74068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.172 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 --2- 172.21.15.64:0/189330268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d740ffd00 0x7f6d7419a660 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 >> 172.21.15.64:0/189330268 conn(0x7f6d740fa130 msgr2=0x7f6d740fc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:01.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 shutdown_connections 2023-07-14T18:25:01.173 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.117+0000 7f6d7a9cd700 1 -- 172.21.15.64:0/189330268 wait complete. 2023-07-14T18:25:01.182 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:25:01.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.258+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1924823949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38100690 msgr2=0x7f5d38102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.312 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.258+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1924823949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38100690 0x7f5d38102a70 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009990 tx=0x7f5d2c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.258+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1924823949 shutdown_connections 2023-07-14T18:25:01.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.258+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1924823949 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d38107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.258+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1924823949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38100690 0x7f5d38102a70 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.313 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1924823949 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38102fb0 0x7f5d38105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1924823949 >> 172.21.15.64:0/1924823949 conn(0x7f5d380fa130 msgr2=0x7f5d380fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:01.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1924823949 shutdown_connections 2023-07-14T18:25:01.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1924823949 wait complete. 2023-07-14T18:25:01.314 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 Processor -- start 2023-07-14T18:25:01.315 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- start start 2023-07-14T18:25:01.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38100690 0x7f5d38100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.317 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d380feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d380ff320 con 0x7f5d38102fb0 2023-07-14T18:25:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d380ff490 con 0x7f5d381058d0 2023-07-14T18:25:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3f60d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3819f990 con 0x7f5d38100690 2023-07-14T18:25:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.259+0000 7f5d3cba8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.318 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:48984/0 (socket says 172.21.15.64:48984) 2023-07-14T18:25:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 -- 172.21.15.64:0/1377934095 learned_addr learned my addr 172.21.15.64:0/1377934095 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3d3a9700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38100690 0x7f5d38100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 -- 172.21.15.64:0/1377934095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38100690 msgr2=0x7f5d38100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3dbaa700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d380feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.319 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38100690 0x7f5d38100000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 -- 172.21.15.64:0/1377934095 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 msgr2=0x7f5d380feb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d380feb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 -- 172.21.15.64:0/1377934095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c0095f0 con 0x7f5d38102fb0 2023-07-14T18:25:01.320 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3dbaa700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d380feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3cba8700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400b530 tx=0x7f5d3400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34011830 con 0x7f5d38102fb0 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5d34011e20 con 0x7f5d38102fb0 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3819fc70 con 0x7f5d38102fb0 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34010dd0 con 0x7f5d38102fb0 2023-07-14T18:25:01.321 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.260+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d381a0180 con 0x7f5d38102fb0 2023-07-14T18:25:01.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.261+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d380fbca0 con 0x7f5d38102fb0 2023-07-14T18:25:01.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.262+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5d34011990 con 0x7f5d38102fb0 2023-07-14T18:25:01.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.263+0000 7f5d2a7fc700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 0x7f5d240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.263+0000 7f5d3d3a9700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 0x7f5d240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.322 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.263+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5d340cc7a0 con 0x7f5d38102fb0 2023-07-14T18:25:01.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.264+0000 7f5d3d3a9700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 0x7f5d240a3690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00b2e0 tx=0x7f5d2c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:01.323 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.268+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5d34095b90 con 0x7f5d38102fb0 2023-07-14T18:25:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.469+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d3802cb00 con 0x7f5d38102fb0 2023-07-14T18:25:01.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.469+0000 7f5d2a7fc700 1 -- 172.21.15.64:0/1377934095 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d340991b0 con 0x7f5d38102fb0 2023-07-14T18:25:01.523 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:25:01.524 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.471+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 msgr2=0x7f5d240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.471+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 0x7f5d240a3690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00b2e0 tx=0x7f5d2c005fb0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.471+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 msgr2=0x7f5d380fe640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.525 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.471+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5d3400b530 tx=0x7f5d3400b8f0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 shutdown_connections 2023-07-14T18:25:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f5d38100690 0x7f5d38100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.526 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f5d240a11e0 0x7f5d240a3690 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5d381058d0 0x7f5d380feb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 --2- 172.21.15.64:0/1377934095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d38102fb0 0x7f5d380fe640 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 >> 172.21.15.64:0/1377934095 conn(0x7f5d380fa130 msgr2=0x7f5d381012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 shutdown_connections 2023-07-14T18:25:01.527 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.472+0000 7f5d3f60d700 1 -- 172.21.15.64:0/1377934095 wait complete. 2023-07-14T18:25:01.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 10 op/s 2023-07-14T18:25:01.536 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:25:01.537 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/593653412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:01.537 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:25:01.537 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/258566223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.538 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:01 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/189330268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.539 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:25:01.669 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.615+0000 7f9038680700 1 -- 172.21.15.64:0/1810498381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 msgr2=0x7f9030106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.669 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.615+0000 7f9038680700 1 --2- 172.21.15.64:0/1810498381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f9030106f80 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f902800b0a0 tx=0x7f902800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.669 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- 172.21.15.64:0/1810498381 shutdown_connections 2023-07-14T18:25:01.670 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- 172.21.15.64:0/1810498381 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300feb40 0x7f90300fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.670 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- 172.21.15.64:0/1810498381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f9030106f80 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.670 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- 172.21.15.64:0/1810498381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300ffd30 0x7f9030100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.670 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- 172.21.15.64:0/1810498381 >> 172.21.15.64:0/1810498381 conn(0x7f90300fa160 msgr2=0x7f90300fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:01.670 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- 172.21.15.64:0/1810498381 shutdown_connections 2023-07-14T18:25:01.671 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- 172.21.15.64:0/1810498381 wait complete. 2023-07-14T18:25:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 Processor -- start 2023-07-14T18:25:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- start start 2023-07-14T18:25:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300feb40 0x7f90300692e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.673 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300ffd30 0x7f903019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903019f640 con 0x7f90301006c0 2023-07-14T18:25:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903019f7b0 con 0x7f90300feb40 2023-07-14T18:25:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.616+0000 7f9038680700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f903019f970 con 0x7f90300ffd30 2023-07-14T18:25:01.674 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.675 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49000/0 (socket says 172.21.15.64:49000) 2023-07-14T18:25:01.675 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 -- 172.21.15.64:0/520281726 learned_addr learned my addr 172.21.15.64:0/520281726 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:01.675 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9035c1b700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300ffd30 0x7f903019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.675 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f903641c700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300feb40 0x7f90300692e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.676 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 -- 172.21.15.64:0/520281726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300ffd30 msgr2=0x7f903019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.676 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300ffd30 0x7f903019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 -- 172.21.15.64:0/520281726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300feb40 msgr2=0x7f90300692e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300feb40 0x7f90300692e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 -- 172.21.15.64:0/520281726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028009d00 con 0x7f90301006c0 2023-07-14T18:25:01.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9036c1d700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f902800bad0 tx=0x7f902800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:01.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800e040 con 0x7f90301006c0 2023-07-14T18:25:01.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9028004e40 con 0x7f90301006c0 2023-07-14T18:25:01.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90280053b0 con 0x7f90301006c0 2023-07-14T18:25:01.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f903019fbf0 con 0x7f90301006c0 2023-07-14T18:25:01.680 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.617+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90301a0070 con 0x7f90301006c0 2023-07-14T18:25:01.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.618+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9030106050 con 0x7f90301006c0 2023-07-14T18:25:01.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.619+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f902801da90 con 0x7f90301006c0 2023-07-14T18:25:01.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.620+0000 7f90277fe700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 0x7f901c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:01.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.620+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f90280ce5a0 con 0x7f90301006c0 2023-07-14T18:25:01.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.620+0000 7f903641c700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 0x7f901c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:01.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.620+0000 7f903641c700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 0x7f901c0a3710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f902c005fd0 tx=0x7f902c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:01.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.623+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9028097990 con 0x7f90301006c0 2023-07-14T18:25:01.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 10 op/s 2023-07-14T18:25:01.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:25:01.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/593653412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:01.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:25:01.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/258566223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.685 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:01 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/189330268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: pgmap v314: 129 pgs: 129 active+clean; 642 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 341 B/s wr, 10 op/s 2023-07-14T18:25:01.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "random", "pin_setting": "0.01", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:25:01.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/593653412' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:01.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T18:25:01.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/258566223' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:01 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/189330268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:01.877 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.824+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f903002caf0 con 0x7f90301006c0 2023-07-14T18:25:01.878 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.825+0000 7f90277fe700 1 -- 172.21.15.64:0/520281726 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f902809afb0 con 0x7f90301006c0 2023-07-14T18:25:01.878 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:25:01.879 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 msgr2=0x7f901c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 0x7f901c0a3710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f902c005fd0 tx=0x7f902c005e50 comp rx=0 tx=0).stop 2023-07-14T18:25:01.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 msgr2=0x7f903019ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:01.880 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f902800bad0 tx=0x7f902800bbb0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 shutdown_connections 2023-07-14T18:25:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f901c0a1260 0x7f901c0a3710 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f90301006c0 0x7f903019ef90 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90300feb40 0x7f90300692e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.881 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.826+0000 7f9038680700 1 --2- 172.21.15.64:0/520281726 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f90300ffd30 0x7f903019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:01.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.827+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 >> 172.21.15.64:0/520281726 conn(0x7f90300fa160 msgr2=0x7f9030103190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:01.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.827+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 shutdown_connections 2023-07-14T18:25:01.882 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.827+0000 7f9038680700 1 -- 172.21.15.64:0/520281726 wait complete. 2023-07-14T18:25:01.892 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:25:02.019 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 -- 172.21.15.64:0/1053583819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa44100690 msgr2=0x7faa44106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.020 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/1053583819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa44100690 0x7faa44106f50 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00b0a0 tx=0x7faa3c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.020 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 -- 172.21.15.64:0/1053583819 shutdown_connections 2023-07-14T18:25:02.020 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/1053583819 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa44100690 0x7faa44106f50 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/1053583819 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa440feb10 0x7faa440fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/1053583819 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440ffd00 0x7faa44100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.021 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 -- 172.21.15.64:0/1053583819 >> 172.21.15.64:0/1053583819 conn(0x7faa440fa130 msgr2=0x7faa440fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:02.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 -- 172.21.15.64:0/1053583819 shutdown_connections 2023-07-14T18:25:02.022 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.967+0000 7faa4bfc0700 1 -- 172.21.15.64:0/1053583819 wait complete. 2023-07-14T18:25:02.026 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 Processor -- start 2023-07-14T18:25:02.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 -- start start 2023-07-14T18:25:02.027 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440feb10 0x7faa44068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.028 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44100690 0x7faa4419cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4419d4f0 con 0x7faa440ffd00 2023-07-14T18:25:02.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4419d660 con 0x7faa44100690 2023-07-14T18:25:02.029 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4bfc0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa4419d820 con 0x7faa440feb10 2023-07-14T18:25:02.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.030 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49012/0 (socket says 172.21.15.64:49012) 2023-07-14T18:25:02.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 -- 172.21.15.64:0/3132996857 learned_addr learned my addr 172.21.15.64:0/3132996857 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:02.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa49d5c700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440feb10 0x7faa44068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4a55d700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44100690 0x7faa4419cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.031 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 -- 172.21.15.64:0/3132996857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440feb10 msgr2=0x7faa44068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440feb10 0x7faa44068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 -- 172.21.15.64:0/3132996857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44100690 msgr2=0x7faa4419cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.032 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44100690 0x7faa4419cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.968+0000 7faa4955b700 1 -- 172.21.15.64:0/3132996857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3c009d00 con 0x7faa440ffd00 2023-07-14T18:25:02.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa4955b700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c920 tx=0x7faa3400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34007b00 con 0x7faa440ffd00 2023-07-14T18:25:02.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faa3400ce70 con 0x7faa440ffd00 2023-07-14T18:25:02.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa4419db00 con 0x7faa440ffd00 2023-07-14T18:25:02.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa34005440 con 0x7faa440ffd00 2023-07-14T18:25:02.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.969+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa4419e010 con 0x7faa440ffd00 2023-07-14T18:25:02.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.970+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa44106100 con 0x7faa440ffd00 2023-07-14T18:25:02.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.971+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7faa340055a0 con 0x7faa440ffd00 2023-07-14T18:25:02.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.971+0000 7faa3affd700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 0x7faa300a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.972+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faa340cc980 con 0x7faa440ffd00 2023-07-14T18:25:02.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.972+0000 7faa49d5c700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 0x7faa300a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.972+0000 7faa49d5c700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 0x7faa300a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faa440ffb60 tx=0x7faa40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:01.976+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faa34095c60 con 0x7faa440ffd00 2023-07-14T18:25:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.190+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7faa4402c8e0 con 0x7faa440ffd00 2023-07-14T18:25:02.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.190+0000 7faa3affd700 1 -- 172.21.15.64:0/3132996857 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+589 (secure 0 0 0) 0x7faa3401d260 con 0x7faa440ffd00 2023-07-14T18:25:02.244 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:02.244 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:02.244 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2932389061 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:02.245 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3783129146 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:02.245 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1024618281 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:02.245 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:02.245 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:02.246 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/377975996 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:02.246 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1532744554 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:02.246 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2890765683 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.192+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 msgr2=0x7faa300a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.192+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 0x7faa300a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7faa440ffb60 tx=0x7faa40009260 comp rx=0 tx=0).stop 2023-07-14T18:25:02.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.192+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 msgr2=0x7faa4419a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.192+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7faa3400c920 tx=0x7faa3400cc30 comp rx=0 tx=0).stop 2023-07-14T18:25:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.192+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 shutdown_connections 2023-07-14T18:25:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa440feb10 0x7faa44068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7faa300a11e0 0x7faa300a3690 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa44100690 0x7faa4419cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 --2- 172.21.15.64:0/3132996857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa440ffd00 0x7faa4419a6d0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 >> 172.21.15.64:0/3132996857 conn(0x7faa440fa130 msgr2=0x7faa44103240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 shutdown_connections 2023-07-14T18:25:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.193+0000 7faa4bfc0700 1 -- 172.21.15.64:0/3132996857 wait complete. 2023-07-14T18:25:02.249 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2023-07-14T18:25:02.258 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2023-07-14T18:25:02.258 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T18:25:02.277 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:25:02.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 -- 172.21.15.64:0/3052014620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 msgr2=0x7f6980106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 --2- 172.21.15.64:0/3052014620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980106f40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f697800b0a0 tx=0x7f697800b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.444 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 -- 172.21.15.64:0/3052014620 shutdown_connections 2023-07-14T18:25:02.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 --2- 172.21.15.64:0/3052014620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980106f40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 --2- 172.21.15.64:0/3052014620 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800feb00 0x7f69800fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 --2- 172.21.15.64:0/3052014620 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800ffcf0 0x7f6980100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 -- 172.21.15.64:0/3052014620 >> 172.21.15.64:0/3052014620 conn(0x7f69800fa100 msgr2=0x7f69800fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:02.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 -- 172.21.15.64:0/3052014620 shutdown_connections 2023-07-14T18:25:02.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.390+0000 7f6985bb1700 1 -- 172.21.15.64:0/3052014620 wait complete. 2023-07-14T18:25:02.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 Processor -- start 2023-07-14T18:25:02.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 -- start start 2023-07-14T18:25:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800feb00 0x7f6980102a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.448 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800ffcf0 0x7f69801010d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980101db0 con 0x7f6980100680 2023-07-14T18:25:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6980101f20 con 0x7f69800ffcf0 2023-07-14T18:25:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f6985bb1700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698019f900 con 0x7f69800feb00 2023-07-14T18:25:02.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f697ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f697ffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49028/0 (socket says 172.21.15.64:49028) 2023-07-14T18:25:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f697ffff700 1 -- 172.21.15.64:0/476286191 learned_addr learned my addr 172.21.15.64:0/476286191 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.391+0000 7f697f7fe700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800feb00 0x7f6980102a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697effd700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800ffcf0 0x7f69801010d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 -- 172.21.15.64:0/476286191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800feb00 msgr2=0x7f6980102a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800feb00 0x7f6980102a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 -- 172.21.15.64:0/476286191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800ffcf0 msgr2=0x7f69801010d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800ffcf0 0x7f69801010d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 -- 172.21.15.64:0/476286191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6978009d00 con 0x7f6980100680 2023-07-14T18:25:02.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697ffff700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f697800b070 tx=0x7f6978004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697800e040 con 0x7f6980100680 2023-07-14T18:25:02.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6978003e90 con 0x7f6980100680 2023-07-14T18:25:02.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698019fb80 con 0x7f6980100680 2023-07-14T18:25:02.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6978004020 con 0x7f6980100680 2023-07-14T18:25:02.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.392+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698019ffb0 con 0x7f6980100680 2023-07-14T18:25:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.393+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698010a1e0 con 0x7f6980100680 2023-07-14T18:25:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.394+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6978003a00 con 0x7f6980100680 2023-07-14T18:25:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.395+0000 7f697cff9700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 0x7f69680a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.395+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f69780cef80 con 0x7f6980100680 2023-07-14T18:25:02.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.395+0000 7f697f7fe700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 0x7f69680a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.395+0000 7f697f7fe700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 0x7f69680a36d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f69800ffb50 tx=0x7f697000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.398+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6978098290 con 0x7f6980100680 2023-07-14T18:25:02.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1377934095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/520281726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.593 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:02 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3132996857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:02.652 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.599+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69801a05c0 con 0x7f6980100680 2023-07-14T18:25:02.653 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.600+0000 7f697cff9700 1 -- 172.21.15.64:0/476286191 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24476 (secure 0 0 0) 0x7f69780243b0 con 0x7f6980100680 2023-07-14T18:25:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:25:02.654 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":87,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:24:07.399732+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:25:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 msgr2=0x7f69680a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 0x7f69680a36d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f69800ffb50 tx=0x7f697000b410 comp rx=0 tx=0).stop 2023-07-14T18:25:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 msgr2=0x7f6980101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.656 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f697800b070 tx=0x7f6978004c80 comp rx=0 tx=0).stop 2023-07-14T18:25:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 shutdown_connections 2023-07-14T18:25:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f69800feb00 0x7f6980102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f69680a1220 0x7f69680a36d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6980100680 0x7f6980101610 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.657 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 --2- 172.21.15.64:0/476286191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f69800ffcf0 0x7f69801010d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 >> 172.21.15.64:0/476286191 conn(0x7f69800fa100 msgr2=0x7f6980067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 shutdown_connections 2023-07-14T18:25:02.658 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.603+0000 7f6985bb1700 1 -- 172.21.15.64:0/476286191 wait complete. 2023-07-14T18:25:02.669 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:19:13.922035+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T18:23:02.854321+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T18:23:04.222100+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T18:23:07.862502+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T18:25:02.671 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T18:25:02.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1377934095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/520281726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:02 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3132996857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:02.800 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 -- 172.21.15.64:0/254407225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1058f0 msgr2=0x7f458c107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 --2- 172.21.15.64:0/254407225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1058f0 0x7f458c107d50 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f458400b0a0 tx=0x7f458400b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.801 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 -- 172.21.15.64:0/254407225 shutdown_connections 2023-07-14T18:25:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 --2- 172.21.15.64:0/254407225 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c1006b0 0x7f458c102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 --2- 172.21.15.64:0/254407225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1058f0 0x7f458c107d50 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 --2- 172.21.15.64:0/254407225 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c102fd0 0x7f458c1053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 -- 172.21.15.64:0/254407225 >> 172.21.15.64:0/254407225 conn(0x7f458c0fa190 msgr2=0x7f458c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:02.802 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 -- 172.21.15.64:0/254407225 shutdown_connections 2023-07-14T18:25:02.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.747+0000 7f45917f6700 1 -- 172.21.15.64:0/254407225 wait complete. 2023-07-14T18:25:02.803 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 Processor -- start 2023-07-14T18:25:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 -- start start 2023-07-14T18:25:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.805 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c102fd0 0x7f458c0fe660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c1058f0 0x7f458c0feba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0ff340 con 0x7f458c1006b0 2023-07-14T18:25:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0ff4b0 con 0x7f458c1058f0 2023-07-14T18:25:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f45917f6700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c19f920 con 0x7f458c102fd0 2023-07-14T18:25:02.806 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f458affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.748+0000 7f458affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49038/0 (socket says 172.21.15.64:49038) 2023-07-14T18:25:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 -- 172.21.15.64:0/3037446857 learned_addr learned my addr 172.21.15.64:0/3037446857 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458b7fe700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c1058f0 0x7f458c0feba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458a7fc700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c102fd0 0x7f458c0fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.807 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 -- 172.21.15.64:0/3037446857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c102fd0 msgr2=0x7f458c0fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.808 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c102fd0 0x7f458c0fe660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.808 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 -- 172.21.15.64:0/3037446857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c1058f0 msgr2=0x7f458c0feba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:02.808 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c1058f0 0x7f458c0feba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:02.808 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 -- 172.21.15.64:0/3037446857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584009d00 con 0x7f458c1006b0 2023-07-14T18:25:02.808 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f458affd700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c9b0 tx=0x7f457c00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c007b00 con 0x7f458c1006b0 2023-07-14T18:25:02.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f457c007c60 con 0x7f458c1006b0 2023-07-14T18:25:02.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c19fc00 con 0x7f458c1006b0 2023-07-14T18:25:02.809 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c018660 con 0x7f458c1006b0 2023-07-14T18:25:02.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.749+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c1a0160 con 0x7f458c1006b0 2023-07-14T18:25:02.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.750+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c0fcd10 con 0x7f458c1006b0 2023-07-14T18:25:02.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.751+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f457c00f450 con 0x7f458c1006b0 2023-07-14T18:25:02.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.752+0000 7f4573fff700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 0x7f45740a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:02.810 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.752+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f457c0ccd30 con 0x7f458c1006b0 2023-07-14T18:25:02.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.752+0000 7f458a7fc700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 0x7f45740a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:02.811 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.752+0000 7f458a7fc700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 0x7f45740a3710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4580009b30 tx=0x7f4580009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:02.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1377934095' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/520281726' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:02.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:02 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3132996857' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:02.812 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.756+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f457c0d1070 con 0x7f458c1006b0 2023-07-14T18:25:03.006 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.953+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f458c02cb20 con 0x7f458c1006b0 2023-07-14T18:25:03.007 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.954+0000 7f4573fff700 1 -- 172.21.15.64:0/3037446857 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f457c096010 con 0x7f458c1006b0 2023-07-14T18:25:03.007 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 1 2023-07-14T18:25:03.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 msgr2=0x7f45740a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.009 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 0x7f45740a3710 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4580009b30 tx=0x7f4580009260 comp rx=0 tx=0).stop 2023-07-14T18:25:03.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 msgr2=0x7f458c100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f457c00c9b0 tx=0x7f457c00ccc0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 shutdown_connections 2023-07-14T18:25:03.010 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f45740a1260 0x7f45740a3710 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f458c1058f0 0x7f458c0feba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f458c1006b0 0x7f458c100020 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 --2- 172.21.15.64:0/3037446857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f458c102fd0 0x7f458c0fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 >> 172.21.15.64:0/3037446857 conn(0x7f458c0fa190 msgr2=0x7f458c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.011 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 shutdown_connections 2023-07-14T18:25:03.012 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:02.956+0000 7f45917f6700 1 -- 172.21.15.64:0/3037446857 wait complete. 2023-07-14T18:25:03.021 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T18:25:03.150 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 -- 172.21.15.64:0/951172084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 msgr2=0x7f9568107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 --2- 172.21.15.64:0/951172084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568107060 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f956000b0a0 tx=0x7f956000b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 -- 172.21.15.64:0/951172084 shutdown_connections 2023-07-14T18:25:03.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 --2- 172.21.15.64:0/951172084 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f9568100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.151 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 --2- 172.21.15.64:0/951172084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568107060 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 --2- 172.21.15.64:0/951172084 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 0x7f95680ff270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 -- 172.21.15.64:0/951172084 >> 172.21.15.64:0/951172084 conn(0x7f95680f9980 msgr2=0x7f95680fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 -- 172.21.15.64:0/951172084 shutdown_connections 2023-07-14T18:25:03.152 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.097+0000 7f956d98a700 1 -- 172.21.15.64:0/951172084 wait complete. 2023-07-14T18:25:03.153 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 Processor -- start 2023-07-14T18:25:03.155 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 -- start start 2023-07-14T18:25:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 0x7f9568194880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f956819be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.156 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956819c400 con 0x7f95680ff7b0 2023-07-14T18:25:03.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956819c540 con 0x7f95680fee00 2023-07-14T18:25:03.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f956d98a700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956819c6d0 con 0x7f9568100240 2023-07-14T18:25:03.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.098+0000 7f95667fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.157 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49054/0 (socket says 172.21.15.64:49054) 2023-07-14T18:25:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 -- 172.21.15.64:0/511392924 learned_addr learned my addr 172.21.15.64:0/511392924 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f9566ffd700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 0x7f9568194880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95677fe700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f956819be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 -- 172.21.15.64:0/511392924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 msgr2=0x7f956819be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.158 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f956819be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 -- 172.21.15.64:0/511392924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 msgr2=0x7f9568194880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 0x7f9568194880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 -- 172.21.15.64:0/511392924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9560009d00 con 0x7f95680ff7b0 2023-07-14T18:25:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95677fe700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f956819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:25:03.159 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f95667fc700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c920 tx=0x7f955c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c007b00 con 0x7f95680ff7b0 2023-07-14T18:25:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f955c00ce70 con 0x7f95680ff7b0 2023-07-14T18:25:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956819c9b0 con 0x7f95680ff7b0 2023-07-14T18:25:03.160 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955c005440 con 0x7f95680ff7b0 2023-07-14T18:25:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.099+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956819cf10 con 0x7f95680ff7b0 2023-07-14T18:25:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.100+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9568106260 con 0x7f95680ff7b0 2023-07-14T18:25:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.101+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f955c0055a0 con 0x7f95680ff7b0 2023-07-14T18:25:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.102+0000 7f956c988700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 0x7f95500a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.161 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.102+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f955c0ccd00 con 0x7f95680ff7b0 2023-07-14T18:25:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.102+0000 7f9566ffd700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 0x7f95500a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.102+0000 7f9566ffd700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 0x7f95500a3710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9558005950 tx=0x7f955800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.162 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.105+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f955c096060 con 0x7f95680ff7b0 2023-07-14T18:25:03.357 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.304+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f956802ca70 con 0x7f95680ff7b0 2023-07-14T18:25:03.358 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.304+0000 7f956c988700 1 -- 172.21.15.64:0/511392924 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f955c01d020 con 0x7f95680ff7b0 2023-07-14T18:25:03.358 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:25:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.306+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 msgr2=0x7f95500a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.359 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.306+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 0x7f95500a3710 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9558005950 tx=0x7f955800b410 comp rx=0 tx=0).stop 2023-07-14T18:25:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.306+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 msgr2=0x7f9568194dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.306+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f955c00c920 tx=0x7f955c00cc30 comp rx=0 tx=0).stop 2023-07-14T18:25:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 shutdown_connections 2023-07-14T18:25:03.360 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f9568100240 0x7f956819be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f95500a1260 0x7f95500a3710 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f95680fee00 0x7f9568194880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 --2- 172.21.15.64:0/511392924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95680ff7b0 0x7f9568194dc0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 >> 172.21.15.64:0/511392924 conn(0x7f95680f9980 msgr2=0x7f95681033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.361 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 shutdown_connections 2023-07-14T18:25:03.362 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.307+0000 7f956d98a700 1 -- 172.21.15.64:0/511392924 wait complete. 2023-07-14T18:25:03.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T18:25:03.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 -- 172.21.15.64:0/3949008286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48940730a0 msgr2=0x7f4894073510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 --2- 172.21.15.64:0/3949008286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48940730a0 0x7f4894073510 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4884009990 tx=0x7f4884009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 -- 172.21.15.64:0/3949008286 shutdown_connections 2023-07-14T18:25:03.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 --2- 172.21.15.64:0/3949008286 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f4894074680 0x7f4894072ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 --2- 172.21.15.64:0/3949008286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f489410c1f0 0x7f489410e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.503 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 --2- 172.21.15.64:0/3949008286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f48940730a0 0x7f4894073510 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 -- 172.21.15.64:0/3949008286 >> 172.21.15.64:0/3949008286 conn(0x7f48940fb260 msgr2=0x7f48940fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 -- 172.21.15.64:0/3949008286 shutdown_connections 2023-07-14T18:25:03.504 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.449+0000 7f489bb58700 1 -- 172.21.15.64:0/3949008286 wait complete. 2023-07-14T18:25:03.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 Processor -- start 2023-07-14T18:25:03.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 -- start start 2023-07-14T18:25:03.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48940730a0 0x7f489419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.506 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4894074680 0x7f489419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a3ac0 con 0x7f489410c1f0 2023-07-14T18:25:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a3c00 con 0x7f4894074680 2023-07-14T18:25:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489bb58700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48941a3d90 con 0x7f48940730a0 2023-07-14T18:25:03.507 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.450+0000 7f489a0f5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49070/0 (socket says 172.21.15.64:49070) 2023-07-14T18:25:03.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 -- 172.21.15.64:0/4256800882 learned_addr learned my addr 172.21.15.64:0/4256800882 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:03.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f48990f3700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4894074680 0x7f489419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f48998f4700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48940730a0 0x7f489419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.508 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 -- 172.21.15.64:0/4256800882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48940730a0 msgr2=0x7f489419bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48940730a0 0x7f489419bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 -- 172.21.15.64:0/4256800882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4894074680 msgr2=0x7f489419c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4894074680 0x7f489419c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 -- 172.21.15.64:0/4256800882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840095f0 con 0x7f489410c1f0 2023-07-14T18:25:03.509 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489a0f5700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f488c00ef30 tx=0x7f488c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c018070 con 0x7f489410c1f0 2023-07-14T18:25:03.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f488c00f040 con 0x7f489410c1f0 2023-07-14T18:25:03.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488c00cc70 con 0x7f489410c1f0 2023-07-14T18:25:03.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48941a4010 con 0x7f489410c1f0 2023-07-14T18:25:03.510 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.451+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48941a44d0 con 0x7f489410c1f0 2023-07-14T18:25:03.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.452+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4894101d40 con 0x7f489410c1f0 2023-07-14T18:25:03.511 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.453+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f488c003a00 con 0x7f489410c1f0 2023-07-14T18:25:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.454+0000 7f488affd700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 0x7f48800a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.512 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.454+0000 7f48998f4700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 0x7f48800a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.454+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f488c0cceb0 con 0x7f489410c1f0 2023-07-14T18:25:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.454+0000 7f48998f4700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 0x7f48800a3710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4890005950 tx=0x7f48900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.457+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f488c096220 con 0x7f489410c1f0 2023-07-14T18:25:03.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:03 smithi144 ceph-mon[98417]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:25:03.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:03 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/476286191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:03.683 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:03 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3037446857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.684 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:03 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/511392924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.657+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f489402ca70 con 0x7f489410c1f0 2023-07-14T18:25:03.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.658+0000 7f488affd700 1 -- 172.21.15.64:0/4256800882 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f488c09f0a0 con 0x7f489410c1f0 2023-07-14T18:25:03.711 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 64 2023-07-14T18:25:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.659+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 msgr2=0x7f48800a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.712 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.659+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 0x7f48800a3710 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4890005950 tx=0x7f48900058e0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.659+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 msgr2=0x7f48941a34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.659+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f488c00ef30 tx=0x7f488c00c3a0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 shutdown_connections 2023-07-14T18:25:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f48940730a0 0x7f489419bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.713 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f48800a1260 0x7f48800a3710 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f489410c1f0 0x7f48941a34f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 --2- 172.21.15.64:0/4256800882 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4894074680 0x7f489419c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 >> 172.21.15.64:0/4256800882 conn(0x7f48940fb260 msgr2=0x7f48940fd640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 shutdown_connections 2023-07-14T18:25:03.714 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.660+0000 7f489bb58700 1 -- 172.21.15.64:0/4256800882 wait complete. 2023-07-14T18:25:03.723 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:03 smithi064 ceph-mon[96720]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:25:03.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:03 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/476286191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:03.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:03 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3037446857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.724 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:03 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/511392924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T18:25:03.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:03 smithi105 ceph-mon[98013]: pgmap v315: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-07-14T18:25:03.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:03 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/476286191' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:03.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:03 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3037446857' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:03 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/511392924' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T18:25:03.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 -- 172.21.15.64:0/489205473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda100feb10 msgr2=0x7fda100fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 --2- 172.21.15.64:0/489205473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda100feb10 0x7fda100fef20 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009990 tx=0x7fd9f8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.855 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 -- 172.21.15.64:0/489205473 shutdown_connections 2023-07-14T18:25:03.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 --2- 172.21.15.64:0/489205473 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda10100690 0x7fda10106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 --2- 172.21.15.64:0/489205473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda100feb10 0x7fda100fef20 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 --2- 172.21.15.64:0/489205473 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100ffd00 0x7fda10100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 -- 172.21.15.64:0/489205473 >> 172.21.15.64:0/489205473 conn(0x7fda100fa130 msgr2=0x7fda100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:03.856 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 -- 172.21.15.64:0/489205473 shutdown_connections 2023-07-14T18:25:03.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 -- 172.21.15.64:0/489205473 wait complete. 2023-07-14T18:25:03.857 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.802+0000 7fda157f0700 1 Processor -- start 2023-07-14T18:25:03.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 -- start start 2023-07-14T18:25:03.859 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 0x7fda101958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda100ffd00 0x7fda10195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019d430 con 0x7fda10100690 2023-07-14T18:25:03.860 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019d570 con 0x7fda100feb10 2023-07-14T18:25:03.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda157f0700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1019d700 con 0x7fda100ffd00 2023-07-14T18:25:03.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49084/0 (socket says 172.21.15.64:49084) 2023-07-14T18:25:03.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 -- 172.21.15.64:0/2975585519 learned_addr learned my addr 172.21.15.64:0/2975585519 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:03.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0effd700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 0x7fda101958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 -- 172.21.15.64:0/2975585519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda100ffd00 msgr2=0x7fda10195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0e7fc700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda100ffd00 0x7fda10195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda100ffd00 0x7fda10195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 -- 172.21.15.64:0/2975585519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 msgr2=0x7fda101958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:03.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 0x7fda101958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 -- 172.21.15.64:0/2975585519 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9f80095f0 con 0x7fda10100690 2023-07-14T18:25:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0effd700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 0x7fda101958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:25:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda0f7fe700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fda08007ee0 tx=0x7fda0800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.803+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08017070 con 0x7fda10100690 2023-07-14T18:25:03.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.804+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fda0800f040 con 0x7fda10100690 2023-07-14T18:25:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.804+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1019d9e0 con 0x7fda10100690 2023-07-14T18:25:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.804+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08005710 con 0x7fda10100690 2023-07-14T18:25:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.804+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1019def0 con 0x7fda10100690 2023-07-14T18:25:03.864 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.804+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda10106cf0 con 0x7fda10100690 2023-07-14T18:25:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.806+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fda08005870 con 0x7fda10100690 2023-07-14T18:25:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.806+0000 7fda07fff700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 0x7fd9fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.806+0000 7fda0effd700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 0x7fd9fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.806+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fda080cc630 con 0x7fda10100690 2023-07-14T18:25:03.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.807+0000 7fda0effd700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 0x7fd9fc0a3690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006010 tx=0x7fd9f8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:03.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:03.811+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fda08095910 con 0x7fda10100690 2023-07-14T18:25:04.063 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.011+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fda1002cb00 con 0x7fda10100690 2023-07-14T18:25:04.064 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.011+0000 7fda07fff700 1 -- 172.21.15.64:0/2975585519 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fda08098f30 con 0x7fda10100690 2023-07-14T18:25:04.065 INFO:teuthology.orchestra.run.smithi064.stdout:pg_num: 32 2023-07-14T18:25:04.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 msgr2=0x7fd9fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 0x7fd9fc0a3690 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8006010 tx=0x7fd9f8005960 comp rx=0 tx=0).stop 2023-07-14T18:25:04.066 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 msgr2=0x7fda1019ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fda08007ee0 tx=0x7fda0800d3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 shutdown_connections 2023-07-14T18:25:04.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fd9fc0a11e0 0x7fd9fc0a3690 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda10100690 0x7fda1019ce60 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.067 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda100feb10 0x7fda101958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 --2- 172.21.15.64:0/2975585519 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fda100ffd00 0x7fda10195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 >> 172.21.15.64:0/2975585519 conn(0x7fda100fa130 msgr2=0x7fda100fc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:04.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 shutdown_connections 2023-07-14T18:25:04.068 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.013+0000 7fda157f0700 1 -- 172.21.15.64:0/2975585519 wait complete. 2023-07-14T18:25:04.078 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T18:25:04.206 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- 172.21.15.64:0/2511321323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 msgr2=0x7f3284100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 --2- 172.21.15.64:0/2511321323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284100180 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f326c009990 tx=0x7f326c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- 172.21.15.64:0/2511321323 shutdown_connections 2023-07-14T18:25:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 --2- 172.21.15.64:0/2511321323 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32841006c0 0x7f3284106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 --2- 172.21.15.64:0/2511321323 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32840feb40 0x7f32840fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.207 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 --2- 172.21.15.64:0/2511321323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284100180 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- 172.21.15.64:0/2511321323 >> 172.21.15.64:0/2511321323 conn(0x7f32840fa160 msgr2=0x7f32840fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:04.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- 172.21.15.64:0/2511321323 shutdown_connections 2023-07-14T18:25:04.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- 172.21.15.64:0/2511321323 wait complete. 2023-07-14T18:25:04.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 Processor -- start 2023-07-14T18:25:04.208 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.153+0000 7f328a714700 1 -- start start 2023-07-14T18:25:04.210 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 0x7f3284194780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:04.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:04.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 0x7f328419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:04.211 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328419c300 con 0x7f32840ffd30 2023-07-14T18:25:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328419c440 con 0x7f32841006c0 2023-07-14T18:25:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f328a714700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328419c5d0 con 0x7f32840feb40 2023-07-14T18:25:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49096/0 (socket says 172.21.15.64:49096) 2023-07-14T18:25:04.212 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 -- 172.21.15.64:0/239304538 learned_addr learned my addr 172.21.15.64:0/239304538 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 -- 172.21.15.64:0/239304538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 msgr2=0x7f3284194780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f3283fff700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 0x7f3284194780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 0x7f3284194780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f3288cb1700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 0x7f328419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:04.213 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 -- 172.21.15.64:0/239304538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 msgr2=0x7f328419bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 0x7f328419bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f32837fe700 1 -- 172.21.15.64:0/239304538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0095f0 con 0x7f32840ffd30 2023-07-14T18:25:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f3283fff700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 0x7f3284194780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:25:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.154+0000 7f3288cb1700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 0x7f328419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:25:04.214 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f32837fe700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f326c005910 tx=0x7f326c0057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:04.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c01d070 con 0x7f32840ffd30 2023-07-14T18:25:04.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f326c022470 con 0x7f32840ffd30 2023-07-14T18:25:04.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328419c850 con 0x7f32840ffd30 2023-07-14T18:25:04.215 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00f6c0 con 0x7f32840ffd30 2023-07-14T18:25:04.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.155+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328419ccd0 con 0x7f32840ffd30 2023-07-14T18:25:04.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.156+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328410a210 con 0x7f32840ffd30 2023-07-14T18:25:04.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.157+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f326c0225e0 con 0x7f32840ffd30 2023-07-14T18:25:04.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.157+0000 7f32817fa700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 0x7f32700a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:04.216 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.158+0000 7f3283fff700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 0x7f32700a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.158+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f326c0ce2e0 con 0x7f32840ffd30 2023-07-14T18:25:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.158+0000 7f3283fff700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 0x7f32700a3710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f32840ffb90 tx=0x7f3274009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:04.217 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.162+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f326c097650 con 0x7f32840ffd30 2023-07-14T18:25:04.431 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.379+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3284105a20 con 0x7f32840ffd30 2023-07-14T18:25:04.432 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.379+0000 7f32817fa700 1 -- 172.21.15.64:0/239304538 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+589 (secure 0 0 0) 0x7f326c09ac70 con 0x7f32840ffd30 2023-07-14T18:25:04.432 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:04.432 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/792928752 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:04.433 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2932389061 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:04.433 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/3783129146 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:04.433 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1024618281 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:04.434 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6801/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:04.434 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:6800/3374567519 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:04.435 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/377975996 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:04.435 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/1532744554 2023-07-15T18:16:37.487452+0000 2023-07-14T18:25:04.435 INFO:teuthology.orchestra.run.smithi064.stdout:172.21.15.64:0/2890765683 2023-07-15T18:17:05.380198+0000 2023-07-14T18:25:04.437 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 msgr2=0x7f32700a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 0x7f32700a3710 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f32840ffb90 tx=0x7f3274009260 comp rx=0 tx=0).stop 2023-07-14T18:25:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 msgr2=0x7f3284194cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:04.438 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f326c005910 tx=0x7f326c0057d0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 shutdown_connections 2023-07-14T18:25:04.439 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f32840feb40 0x7f3284194780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f32700a1260 0x7f32700a3710 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.440 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32841006c0 0x7f328419bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 --2- 172.21.15.64:0/239304538 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f32840ffd30 0x7f3284194cc0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:04.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 >> 172.21.15.64:0/239304538 conn(0x7f32840fa160 msgr2=0x7f32840fc390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:04.441 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 shutdown_connections 2023-07-14T18:25:04.442 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:04.381+0000 7f328a714700 1 -- 172.21.15.64:0/239304538 wait complete. 2023-07-14T18:25:04.442 INFO:teuthology.orchestra.run.smithi064.stderr:listed 10 entries 2023-07-14T18:25:04.447 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi064.front.sepia.ceph.com... 2023-07-14T18:25:04.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:25:04.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-07-14T18:25:04.484 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi064.stderr:ceph-fuse[147999]: fuse finished with error 0 and tester_r 0 2023-07-14T18:25:04.485 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T18:25:04.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4256800882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:04.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:04 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2975585519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:04.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4256800882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:04.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:04 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2975585519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:04.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4256800882' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T18:25:04.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:04 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2975585519' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T18:25:05.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:05 smithi105 ceph-mon[98013]: pgmap v316: 129 pgs: 129 active+clean; 649 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2023-07-14T18:25:05.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:05 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/239304538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:05.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:05 smithi064 ceph-mon[96720]: pgmap v316: 129 pgs: 129 active+clean; 649 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2023-07-14T18:25:05.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:05 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/239304538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:05.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:05 smithi144 ceph-mon[98417]: pgmap v316: 129 pgs: 129 active+clean; 649 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2023-07-14T18:25:05.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:05 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/239304538' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T18:25:07.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-07-14T18:25:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-07-14T18:25:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T18:25:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2023-07-14T18:25:07.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2023-07-14T18:25:07.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:07 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2023-07-14T18:25:07.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-07-14T18:25:07.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-07-14T18:25:07.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T18:25:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2023-07-14T18:25:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2023-07-14T18:25:07.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:07 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2023-07-14T18:25:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: pgmap v317: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2023-07-14T18:25:07.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]: dispatch 2023-07-14T18:25:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2023-07-14T18:25:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]: dispatch 2023-07-14T18:25:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2023-07-14T18:25:07.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:07 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2023-07-14T18:25:08.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-07-14T18:25:08.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T18:25:08.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2023-07-14T18:25:08.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2023-07-14T18:25:08.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2023-07-14T18:25:08.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:08 smithi105 ceph-mon[98013]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:25:08.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-07-14T18:25:08.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T18:25:08.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2023-07-14T18:25:08.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2023-07-14T18:25:08.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2023-07-14T18:25:08.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:08 smithi064 ceph-mon[96720]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:25:08.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.6", "id": [3, 7]}]': finished 2023-07-14T18:25:08.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2023-07-14T18:25:08.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.d", "id": [1, 9]}]': finished 2023-07-14T18:25:08.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2023-07-14T18:25:08.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2023-07-14T18:25:08.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:08 smithi144 ceph-mon[98417]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T18:25:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:09 smithi105 ceph-mon[98013]: pgmap v319: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-07-14T18:25:09.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:09 smithi105 ceph-mon[98013]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:25:09.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:09 smithi064 ceph-mon[96720]: pgmap v319: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-07-14T18:25:09.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:09 smithi064 ceph-mon[96720]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:25:09.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:09 smithi144 ceph-mon[98417]: pgmap v319: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 13 op/s 2023-07-14T18:25:09.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:09 smithi144 ceph-mon[98417]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T18:25:10.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:25:10.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:25:10.588 DEBUG:teuthology.orchestra.run.smithi064:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T18:25:10.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:25:10.612 DEBUG:teuthology.orchestra.run.smithi064:> ip netns list 2023-07-14T18:25:10.673 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T18:25:10.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:25:10.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T18:25:10.755 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:10 smithi064 ceph-mon[96720]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-14T18:25:10.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T18:25:10.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo ip link delete ceph-brx 2023-07-14T18:25:10.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:10 smithi105 ceph-mon[98013]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-14T18:25:10.887 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T18:25:10.898 INFO:tasks.cephadm:Teardown begin 2023-07-14T18:25:10.899 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T18:25:10.899 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:25:10.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:10 smithi144 ceph-mon[98417]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-07-14T18:25:11.033 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- 172.21.15.64:0/1480664120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 msgr2=0x7f9764106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 --2- 172.21.15.64:0/1480664120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f9764106f40 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f975c00b0a0 tx=0x7f975c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- 172.21.15.64:0/1480664120 shutdown_connections 2023-07-14T18:25:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 --2- 172.21.15.64:0/1480664120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f9764106f40 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.034 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 --2- 172.21.15.64:0/1480664120 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640feb00 0x7f97640fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 --2- 172.21.15.64:0/1480664120 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640ffcf0 0x7f9764100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- 172.21.15.64:0/1480664120 >> 172.21.15.64:0/1480664120 conn(0x7f97640fa100 msgr2=0x7f97640fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:11.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- 172.21.15.64:0/1480664120 shutdown_connections 2023-07-14T18:25:11.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- 172.21.15.64:0/1480664120 wait complete. 2023-07-14T18:25:11.035 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 Processor -- start 2023-07-14T18:25:11.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.980+0000 7f976a104700 1 -- start start 2023-07-14T18:25:11.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640feb00 0x7f9764194850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640ffcf0 0x7f9764194d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.036 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976419c3d0 con 0x7f9764100680 2023-07-14T18:25:11.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976419c510 con 0x7f97640ffcf0 2023-07-14T18:25:11.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976419c6a0 con 0x7f97640feb00 2023-07-14T18:25:11.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.037 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49110/0 (socket says 172.21.15.64:49110) 2023-07-14T18:25:11.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 -- 172.21.15.64:0/4126590435 learned_addr learned my addr 172.21.15.64:0/4126590435 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:11.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f97637fe700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640feb00 0x7f9764194850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9762ffd700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640ffcf0 0x7f9764194d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 -- 172.21.15.64:0/4126590435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640feb00 msgr2=0x7f9764194850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.038 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640feb00 0x7f9764194850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 -- 172.21.15.64:0/4126590435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640ffcf0 msgr2=0x7f9764194d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640ffcf0 0x7f9764194d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 -- 172.21.15.64:0/4126590435 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975c009d00 con 0x7f9764100680 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9763fff700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f975c009fd0 tx=0x7f975c00ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c00e040 con 0x7f9764100680 2023-07-14T18:25:11.039 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f975c003c00 con 0x7f9764100680 2023-07-14T18:25:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976419c920 con 0x7f9764100680 2023-07-14T18:25:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.981+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975c01b6c0 con 0x7f9764100680 2023-07-14T18:25:11.040 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.982+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f976419cd40 con 0x7f9764100680 2023-07-14T18:25:11.041 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.982+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9764105ff0 con 0x7f9764100680 2023-07-14T18:25:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.983+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f975c01d470 con 0x7f9764100680 2023-07-14T18:25:11.042 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.984+0000 7f9760ff9700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 0x7f974c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.984+0000 7f97637fe700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 0x7f974c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.984+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f975c0ce620 con 0x7f9764100680 2023-07-14T18:25:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.985+0000 7f97637fe700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 0x7f974c0a36d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f97640ffb50 tx=0x7f9754009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.043 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:10.988+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f975c097900 con 0x7f9764100680 2023-07-14T18:25:11.291 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.239+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f976402c8e0 con 0x7f9764100680 2023-07-14T18:25:11.292 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.240+0000 7f9760ff9700 1 -- 172.21.15.64:0/4126590435 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10891 (secure 0 0 0) 0x7f975c09af20 con 0x7f9764100680 2023-07-14T18:25:11.293 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:25:11.294 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:24:59.357577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:25:11.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 msgr2=0x7f974c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.295 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 0x7f974c0a36d0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f97640ffb50 tx=0x7f9754009260 comp rx=0 tx=0).stop 2023-07-14T18:25:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 msgr2=0x7f976419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f975c009fd0 tx=0x7f975c00ba50 comp rx=0 tx=0).stop 2023-07-14T18:25:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 shutdown_connections 2023-07-14T18:25:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f97640feb00 0x7f9764194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.296 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f974c0a1220 0x7f974c0a36d0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9764100680 0x7f976419be00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 --2- 172.21.15.64:0/4126590435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f97640ffcf0 0x7f9764194d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 >> 172.21.15.64:0/4126590435 conn(0x7f97640fa100 msgr2=0x7f9764103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 shutdown_connections 2023-07-14T18:25:11.297 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.243+0000 7f976a104700 1 -- 172.21.15.64:0/4126590435 wait complete. 2023-07-14T18:25:11.298 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 28 2023-07-14T18:25:11.308 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T18:25:11.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 -- 172.21.15.64:0/3934452814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec0fdc50 msgr2=0x7f27ec0fe0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.445 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/3934452814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec0fdc50 0x7f27ec0fe0a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f27dc009990 tx=0x7f27dc009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 -- 172.21.15.64:0/3934452814 shutdown_connections 2023-07-14T18:25:11.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/3934452814 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 0x7f27ec1081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/3934452814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec1011e0 0x7f27ec1015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.446 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/3934452814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec0fdc50 0x7f27ec0fe0a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 -- 172.21.15.64:0/3934452814 >> 172.21.15.64:0/3934452814 conn(0x7f27ec0f98a0 msgr2=0x7f27ec0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:11.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 -- 172.21.15.64:0/3934452814 shutdown_connections 2023-07-14T18:25:11.447 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.393+0000 7f27f3a8c700 1 -- 172.21.15.64:0/3934452814 wait complete. 2023-07-14T18:25:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 Processor -- start 2023-07-14T18:25:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 -- start start 2023-07-14T18:25:11.449 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 0x7f27ec1957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 0x7f27ec195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19d370 con 0x7f27ec1011e0 2023-07-14T18:25:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19d4b0 con 0x7f27ec0fdc50 2023-07-14T18:25:11.450 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f3a8c700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec19d640 con 0x7f27ec0fe5e0 2023-07-14T18:25:11.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49124/0 (socket says 172.21.15.64:49124) 2023-07-14T18:25:11.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 -- 172.21.15.64:0/1923975939 learned_addr learned my addr 172.21.15.64:0/1923975939 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:11.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f1828700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 0x7f27ec1957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.451 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f1027700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 0x7f27ec195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 -- 172.21.15.64:0/1923975939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 msgr2=0x7f27ec195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 0x7f27ec195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 -- 172.21.15.64:0/1923975939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 msgr2=0x7f27ec1957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 0x7f27ec1957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.452 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f2029700 1 -- 172.21.15.64:0/1923975939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0095f0 con 0x7f27ec1011e0 2023-07-14T18:25:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.394+0000 7f27f1828700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 0x7f27ec1957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:25:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27f2029700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f27e4007ee0 tx=0x7f27e400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4017070 con 0x7f27ec1011e0 2023-07-14T18:25:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27e400f040 con 0x7f27ec1011e0 2023-07-14T18:25:11.453 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec19d920 con 0x7f27ec1011e0 2023-07-14T18:25:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4005570 con 0x7f27ec1011e0 2023-07-14T18:25:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.395+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec19de80 con 0x7f27ec1011e0 2023-07-14T18:25:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.396+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec107310 con 0x7f27ec1011e0 2023-07-14T18:25:11.454 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.397+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f27e400db20 con 0x7f27ec1011e0 2023-07-14T18:25:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.398+0000 7f27e2ffd700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 0x7f27d80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.398+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f27e40cc4c0 con 0x7f27ec1011e0 2023-07-14T18:25:11.455 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.398+0000 7f27f1828700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 0x7f27d80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.398+0000 7f27f1828700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 0x7f27d80a3750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0674c0 tx=0x7f27e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.456 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.403+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27e4095710 con 0x7f27ec1011e0 2023-07-14T18:25:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:11 smithi064 ceph-mon[96720]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2023-07-14T18:25:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:11 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:25:11.592 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:11 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/4126590435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:11.704 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.652+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27ec02c8e0 con 0x7f27ec1011e0 2023-07-14T18:25:11.705 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.653+0000 7f27e2ffd700 1 -- 172.21.15.64:0/1923975939 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+10891 (secure 0 0 0) 0x7f27e4098d30 con 0x7f27ec1011e0 2023-07-14T18:25:11.706 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:25:11.706 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14526,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.144:6833/4249483747","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6832","nonce":4249483747},{"type":"v1","addr":"172.21.15.144:6833","nonce":4249483747}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":14538,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6833/3187736260","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6832","nonce":3187736260},{"type":"v1","addr":"172.21.15.105:6833","nonce":3187736260}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T18:23:04.865157+0000","modified":"2023-07-14T18:24:59.357577+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24545,"mds_1":24508,"mds_2":24500,"mds_3":24487,"mds_4":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14550":{"gid":14550,"name":"j","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/2756017867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":2756017867},{"type":"v1","addr":"172.21.15.64:6835","nonce":2756017867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14556":{"gid":14556,"name":"d","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6837/1035088143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":1035088143},{"type":"v1","addr":"172.21.15.64:6837","nonce":1035088143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14616":{"gid":14616,"name":"g","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6839/3820392799","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":3820392799},{"type":"v1","addr":"172.21.15.64:6839","nonce":3820392799}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14628":{"gid":14628,"name":"a","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6841/146209694","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6840","nonce":146209694},{"type":"v1","addr":"172.21.15.64:6841","nonce":146209694}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.105:6835/2365162845","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6834","nonce":2365162845},{"type":"v1","addr":"172.21.15.105:6835","nonce":2365162845}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"f","rank":4,"incarnation":25,"state":"up:active","state_seq":12,"addr":"172.21.15.144:6835/4223527896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6834","nonce":4223527896},{"type":"v1","addr":"172.21.15.144:6835","nonce":4223527896}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"c","rank":3,"incarnation":23,"state":"up:active","state_seq":11,"addr":"172.21.15.144:6837/1582348642","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6836","nonce":1582348642},{"type":"v1","addr":"172.21.15.144:6837","nonce":1582348642}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24500":{"gid":24500,"name":"h","rank":2,"incarnation":21,"state":"up:active","state_seq":10,"addr":"172.21.15.105:6837/2898950423","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6836","nonce":2898950423},{"type":"v1","addr":"172.21.15.105:6837","nonce":2898950423}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24508":{"gid":24508,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":9,"addr":"172.21.15.144:6839/4051776603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6838","nonce":4051776603},{"type":"v1","addr":"172.21.15.144:6839","nonce":4051776603}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24545":{"gid":24545,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.105:6839/2402376208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6838","nonce":2402376208},{"type":"v1","addr":"172.21.15.105:6839","nonce":2402376208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T18:25:11.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.655+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 msgr2=0x7f27d80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.655+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 0x7f27d80a3750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0674c0 tx=0x7f27e800b410 comp rx=0 tx=0).stop 2023-07-14T18:25:11.707 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.655+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 msgr2=0x7f27ec19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.708 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.655+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f27e4007ee0 tx=0x7f27e400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 shutdown_connections 2023-07-14T18:25:11.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f27d80a12a0 0x7f27d80a3750 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f27ec1011e0 0x7f27ec19cda0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27ec0fdc50 0x7f27ec1957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.709 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 --2- 172.21.15.64:0/1923975939 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f27ec0fe5e0 0x7f27ec195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 >> 172.21.15.64:0/1923975939 conn(0x7f27ec0f98a0 msgr2=0x7f27ec104450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:11.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 shutdown_connections 2023-07-14T18:25:11.710 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.656+0000 7f27f3a8c700 1 -- 172.21.15.64:0/1923975939 wait complete. 2023-07-14T18:25:11.710 INFO:teuthology.orchestra.run.smithi064.stderr:dumped fsmap epoch 28 2023-07-14T18:25:11.722 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T18:25:11.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:11 smithi105 ceph-mon[98013]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2023-07-14T18:25:11.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:11 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:25:11.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:11 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/4126590435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 -- 172.21.15.64:0/2984216472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484103790 msgr2=0x7fc484103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.861 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 --2- 172.21.15.64:0/2984216472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484103790 0x7fc484103c00 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00b0a0 tx=0x7fc47c00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 -- 172.21.15.64:0/2984216472 shutdown_connections 2023-07-14T18:25:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 --2- 172.21.15.64:0/2984216472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484103790 0x7fc484103c00 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 --2- 172.21.15.64:0/2984216472 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc4840fe470 0x7fc484100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.862 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 --2- 172.21.15.64:0/2984216472 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc484100dd0 0x7fc484103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 -- 172.21.15.64:0/2984216472 >> 172.21.15.64:0/2984216472 conn(0x7fc4840fa100 msgr2=0x7fc4840fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:11.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 -- 172.21.15.64:0/2984216472 shutdown_connections 2023-07-14T18:25:11.863 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.809+0000 7fc488ee7700 1 -- 172.21.15.64:0/2984216472 wait complete. 2023-07-14T18:25:11.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 Processor -- start 2023-07-14T18:25:11.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 -- start start 2023-07-14T18:25:11.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4840fe470 0x7fc48419bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.865 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484103790 0x7fc4841a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4841a3ac0 con 0x7fc484100dd0 2023-07-14T18:25:11.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4841a3c00 con 0x7fc484103790 2023-07-14T18:25:11.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc488ee7700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4841a3d90 con 0x7fc4840fe470 2023-07-14T18:25:11.866 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc481d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc481d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49132/0 (socket says 172.21.15.64:49132) 2023-07-14T18:25:11.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc481d9b700 1 -- 172.21.15.64:0/1151154533 learned_addr learned my addr 172.21.15.64:0/1151154533 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:11.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc48259c700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4840fe470 0x7fc48419bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.810+0000 7fc482d9d700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484103790 0x7fc4841a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.867 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 -- 172.21.15.64:0/1151154533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4840fe470 msgr2=0x7fc48419bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4840fe470 0x7fc48419bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 -- 172.21.15.64:0/1151154533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484103790 msgr2=0x7fc4841a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:11.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484103790 0x7fc4841a34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:11.868 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 -- 172.21.15.64:0/1151154533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc47c009d00 con 0x7fc484100dd0 2023-07-14T18:25:11.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc481d9b700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc47800d950 tx=0x7fc47800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800f890 con 0x7fc484100dd0 2023-07-14T18:25:11.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc47800fe80 con 0x7fc484100dd0 2023-07-14T18:25:11.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4841a4070 con 0x7fc484100dd0 2023-07-14T18:25:11.869 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47800e3b0 con 0x7fc484100dd0 2023-07-14T18:25:11.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.811+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4841a4580 con 0x7fc484100dd0 2023-07-14T18:25:11.870 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.812+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4840fbc70 con 0x7fc484100dd0 2023-07-14T18:25:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.814+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc478010460 con 0x7fc484100dd0 2023-07-14T18:25:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.814+0000 7fc4737fe700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 0x7fc46c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.815+0000 7fc48259c700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 0x7fc46c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.815+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9165+0+0 (secure 0 0 0) 0x7fc4780ccbb0 con 0x7fc484100dd0 2023-07-14T18:25:11.871 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.815+0000 7fc48259c700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 0x7fc46c0a36d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc474005950 tx=0x7fc47400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:11.873 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:11.817+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc478095e00 con 0x7fc484100dd0 2023-07-14T18:25:11.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:11 smithi144 ceph-mon[98417]: pgmap v321: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 2.6 KiB/s wr, 16 op/s 2023-07-14T18:25:11.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:11 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:25:11.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:11 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/4126590435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.087 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.035+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc48402cb30 con 0x7fc484100dd0 2023-07-14T18:25:12.089 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.037+0000 7fc4737fe700 1 -- 172.21.15.64:0/1151154533 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24708 (secure 0 0 0) 0x7fc478016090 con 0x7fc484100dd0 2023-07-14T18:25:12.090 INFO:teuthology.orchestra.run.smithi064.stdout: 2023-07-14T18:25:12.090 INFO:teuthology.orchestra.run.smithi064.stdout:{"epoch":89,"fsid":"5441c8fe-2272-11ee-9b33-001a4aab830c","created":"2023-07-14T18:16:04.094382+0000","modified":"2023-07-14T18:25:08.393060+0000","last_up_change":"2023-07-14T18:22:23.207855+0000","last_in_change":"2023-07-14T18:22:05.791515+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:19:13.922035+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T18:23:02.854321+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T18:23:04.222100+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T18:23:07.862502+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a183b85a-fd4d-4bc0-ad4f-40a7e3761093","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6803","nonce":1693019669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6805","nonce":1693019669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6809","nonce":1693019669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1693019669},{"type":"v1","addr":"172.21.15.64:6807","nonce":1693019669}]},"public_addr":"172.21.15.64:6803/1693019669","cluster_addr":"172.21.15.64:6805/1693019669","heartbeat_back_addr":"172.21.15.64:6809/1693019669","heartbeat_front_addr":"172.21.15.64:6807/1693019669","state":["exists","up"]},{"osd":1,"uuid":"195865a8-38c0-48ce-afbd-447f46cc517a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6801","nonce":1803665437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6803","nonce":1803665437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6807","nonce":1803665437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":1803665437},{"type":"v1","addr":"172.21.15.105:6805","nonce":1803665437}]},"public_addr":"172.21.15.105:6801/1803665437","cluster_addr":"172.21.15.105:6803/1803665437","heartbeat_back_addr":"172.21.15.105:6807/1803665437","heartbeat_front_addr":"172.21.15.105:6805/1803665437","state":["exists","up"]},{"osd":2,"uuid":"c631140b-b28e-41b6-9d45-259e15845aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6801","nonce":1093978507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6803","nonce":1093978507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6807","nonce":1093978507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":1093978507},{"type":"v1","addr":"172.21.15.144:6805","nonce":1093978507}]},"public_addr":"172.21.15.144:6801/1093978507","cluster_addr":"172.21.15.144:6803/1093978507","heartbeat_back_addr":"172.21.15.144:6807/1093978507","heartbeat_front_addr":"172.21.15.144:6805/1093978507","state":["exists","up"]},{"osd":3,"uuid":"0fe2d4b3-b366-4899-8979-87184b007cb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6811","nonce":2893239811}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6813","nonce":2893239811}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6817","nonce":2893239811}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":2893239811},{"type":"v1","addr":"172.21.15.64:6815","nonce":2893239811}]},"public_addr":"172.21.15.64:6811/2893239811","cluster_addr":"172.21.15.64:6813/2893239811","heartbeat_back_addr":"172.21.15.64:6817/2893239811","heartbeat_front_addr":"172.21.15.64:6815/2893239811","state":["exists","up"]},{"osd":4,"uuid":"f57e7434-c154-40a8-8b1e-13901e8bc9e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6809","nonce":1784745142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6811","nonce":1784745142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6815","nonce":1784745142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":1784745142},{"type":"v1","addr":"172.21.15.105:6813","nonce":1784745142}]},"public_addr":"172.21.15.105:6809/1784745142","cluster_addr":"172.21.15.105:6811/1784745142","heartbeat_back_addr":"172.21.15.105:6815/1784745142","heartbeat_front_addr":"172.21.15.105:6813/1784745142","state":["exists","up"]},{"osd":5,"uuid":"d9bbd5f1-ec36-41d1-83ac-69abdcfda6bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6809","nonce":681164415}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6811","nonce":681164415}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6815","nonce":681164415}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":681164415},{"type":"v1","addr":"172.21.15.144:6813","nonce":681164415}]},"public_addr":"172.21.15.144:6809/681164415","cluster_addr":"172.21.15.144:6811/681164415","heartbeat_back_addr":"172.21.15.144:6815/681164415","heartbeat_front_addr":"172.21.15.144:6813/681164415","state":["exists","up"]},{"osd":6,"uuid":"13faa2e5-0f41-43f8-a486-b5404b338724","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6819","nonce":394567036}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6821","nonce":394567036}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6825","nonce":394567036}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":394567036},{"type":"v1","addr":"172.21.15.64:6823","nonce":394567036}]},"public_addr":"172.21.15.64:6819/394567036","cluster_addr":"172.21.15.64:6821/394567036","heartbeat_back_addr":"172.21.15.64:6825/394567036","heartbeat_front_addr":"172.21.15.64:6823/394567036","state":["exists","up"]},{"osd":7,"uuid":"cecbd2fb-20ee-4dd8-b9a0-4a92030de5b3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6817","nonce":554937712}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6819","nonce":554937712}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6823","nonce":554937712}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":554937712},{"type":"v1","addr":"172.21.15.105:6821","nonce":554937712}]},"public_addr":"172.21.15.105:6817/554937712","cluster_addr":"172.21.15.105:6819/554937712","heartbeat_back_addr":"172.21.15.105:6823/554937712","heartbeat_front_addr":"172.21.15.105:6821/554937712","state":["exists","up"]},{"osd":8,"uuid":"f001163e-106e-403e-94c6-81f02ae052ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6817","nonce":1466260648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6819","nonce":1466260648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6823","nonce":1466260648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1466260648},{"type":"v1","addr":"172.21.15.144:6821","nonce":1466260648}]},"public_addr":"172.21.15.144:6817/1466260648","cluster_addr":"172.21.15.144:6819/1466260648","heartbeat_back_addr":"172.21.15.144:6823/1466260648","heartbeat_front_addr":"172.21.15.144:6821/1466260648","state":["exists","up"]},{"osd":9,"uuid":"5efb2271-1e04-4bb2-9651-52760025b9ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6827","nonce":2478630818}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6829","nonce":2478630818}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6833","nonce":2478630818}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":2478630818},{"type":"v1","addr":"172.21.15.64:6831","nonce":2478630818}]},"public_addr":"172.21.15.64:6827/2478630818","cluster_addr":"172.21.15.64:6829/2478630818","heartbeat_back_addr":"172.21.15.64:6833/2478630818","heartbeat_front_addr":"172.21.15.64:6831/2478630818","state":["exists","up"]},{"osd":10,"uuid":"dc02c6f1-4ca9-4c42-a5fa-a68e3e8d2dad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6825","nonce":1056583141}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6827","nonce":1056583141}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6830","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6831","nonce":1056583141}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6828","nonce":1056583141},{"type":"v1","addr":"172.21.15.105:6829","nonce":1056583141}]},"public_addr":"172.21.15.105:6825/1056583141","cluster_addr":"172.21.15.105:6827/1056583141","heartbeat_back_addr":"172.21.15.105:6831/1056583141","heartbeat_front_addr":"172.21.15.105:6829/1056583141","state":["exists","up"]},{"osd":11,"uuid":"8b6b9c82-975a-4d23-9b9c-8f4a8f9a1599","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6825","nonce":852142335}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6827","nonce":852142335}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6831","nonce":852142335}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":852142335},{"type":"v1","addr":"172.21.15.144:6829","nonce":852142335}]},"public_addr":"172.21.15.144:6825/852142335","cluster_addr":"172.21.15.144:6827/852142335","heartbeat_back_addr":"172.21.15.144:6831/852142335","heartbeat_front_addr":"172.21.15.144:6829/852142335","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:28.195333+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:18:49.122237+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:10.810651+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:32.286350+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:19:52.738066+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:14.998295+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:36.874365+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:20:57.381846+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:18.672367+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:21:39.928314+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:00.556846+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:22:21.700172+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.4","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11}]},{"pgid":"3.6","mappings":[{"from":3,"to":7}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.d","mappings":[{"from":1,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":7}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.37","mappings":[{"from":5,"to":8}]},{"pgid":"4.e","mappings":[{"from":9,"to":11}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.1d","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.64:6801/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:6800/792928752":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/2932389061":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/3783129146":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/1024618281":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6801/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:6800/3374567519":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/377975996":"2023-07-15T18:17:05.380198+0000","172.21.15.64:0/1532744554":"2023-07-15T18:16:37.487452+0000","172.21.15.64:0/2890765683":"2023-07-15T18:17:05.380198+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T18:25:12.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 msgr2=0x7fc46c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.092 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 0x7fc46c0a36d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc474005950 tx=0x7fc47400b410 comp rx=0 tx=0).stop 2023-07-14T18:25:12.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 msgr2=0x7fc48419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.093 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fc47800d950 tx=0x7fc47800dc60 comp rx=0 tx=0).stop 2023-07-14T18:25:12.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 shutdown_connections 2023-07-14T18:25:12.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7fc4840fe470 0x7fc48419bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.094 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fc46c0a1220 0x7fc46c0a36d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc484103790 0x7fc4841a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.095 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 --2- 172.21.15.64:0/1151154533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc484100dd0 0x7fc48419c480 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 >> 172.21.15.64:0/1151154533 conn(0x7fc4840fa100 msgr2=0x7fc4840ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 shutdown_connections 2023-07-14T18:25:12.096 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.039+0000 7fc488ee7700 1 -- 172.21.15.64:0/1151154533 wait complete. 2023-07-14T18:25:12.104 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T18:25:12.240 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 -- 172.21.15.64:0/521110337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40ffd00 msgr2=0x7f10b4100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 --2- 172.21.15.64:0/521110337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40ffd00 0x7f10b4100150 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f109c009990 tx=0x7f109c009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.241 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 -- 172.21.15.64:0/521110337 shutdown_connections 2023-07-14T18:25:12.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 --2- 172.21.15.64:0/521110337 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b4100690 0x7f10b4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 --2- 172.21.15.64:0/521110337 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b40feb10 0x7f10b40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.242 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 --2- 172.21.15.64:0/521110337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40ffd00 0x7f10b4100150 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 -- 172.21.15.64:0/521110337 >> 172.21.15.64:0/521110337 conn(0x7f10b40fa130 msgr2=0x7f10b40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:12.243 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 -- 172.21.15.64:0/521110337 shutdown_connections 2023-07-14T18:25:12.244 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.188+0000 7f10b8b09700 1 -- 172.21.15.64:0/521110337 wait complete. 2023-07-14T18:25:12.247 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 Processor -- start 2023-07-14T18:25:12.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 -- start start 2023-07-14T18:25:12.248 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b40ffd00 0x7f10b4195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.249 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b4100690 0x7f10b419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b419d3e0 con 0x7f10b40feb10 2023-07-14T18:25:12.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b419d520 con 0x7f10b4100690 2023-07-14T18:25:12.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.189+0000 7f10b8b09700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b419d6b0 con 0x7f10b40ffd00 2023-07-14T18:25:12.250 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49144/0 (socket says 172.21.15.64:49144) 2023-07-14T18:25:12.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 -- 172.21.15.64:0/569465764 learned_addr learned my addr 172.21.15.64:0/569465764 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:12.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b2d9d700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b4100690 0x7f10b419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10a9d9b700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b40ffd00 0x7f10b4195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.251 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 -- 172.21.15.64:0/569465764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b40ffd00 msgr2=0x7f10b4195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b40ffd00 0x7f10b4195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 -- 172.21.15.64:0/569465764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b4100690 msgr2=0x7f10b419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.252 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b4100690 0x7f10b419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 -- 172.21.15.64:0/569465764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f109c0095f0 con 0x7f10b40feb10 2023-07-14T18:25:12.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b259c700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f10a400d980 tx=0x7f10a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:12.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a400f890 con 0x7f10b40feb10 2023-07-14T18:25:12.253 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b419d990 con 0x7f10b40feb10 2023-07-14T18:25:12.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f10a400fe80 con 0x7f10b40feb10 2023-07-14T18:25:12.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4019360 con 0x7f10b40feb10 2023-07-14T18:25:12.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.190+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b419def0 con 0x7f10b40feb10 2023-07-14T18:25:12.254 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.191+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b410a190 con 0x7f10b40feb10 2023-07-14T18:25:12.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.192+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f10a400f9f0 con 0x7f10b40feb10 2023-07-14T18:25:12.255 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.193+0000 7f10abfff700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 0x7f10a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.193+0000 7f10a9d9b700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 0x7f10a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.256 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.193+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9165+0+0 (secure 0 0 0) 0x7f10a40ccb90 con 0x7f10b40feb10 2023-07-14T18:25:12.257 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.194+0000 7f10a9d9b700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 0x7f10a00a3690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f109c00b2e0 tx=0x7f109c004000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:12.258 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.197+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f10a4095de0 con 0x7f10b40feb10 2023-07-14T18:25:12.502 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.450+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f10b410a350 con 0x7f10b40feb10 2023-07-14T18:25:12.513 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.461+0000 7f10abfff700 1 -- 172.21.15.64:0/569465764 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v29) v1 ==== 161+0+0 (secure 0 0 0) 0x7f10a4020090 con 0x7f10b40feb10 2023-07-14T18:25:12.517 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.464+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 msgr2=0x7f10a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.464+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 0x7f10a00a3690 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f109c00b2e0 tx=0x7f109c004000 comp rx=0 tx=0).stop 2023-07-14T18:25:12.518 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.464+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 msgr2=0x7f10b4195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.464+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f10a400d980 tx=0x7f10a400dc90 comp rx=0 tx=0).stop 2023-07-14T18:25:12.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 shutdown_connections 2023-07-14T18:25:12.519 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f10a00a11e0 0x7f10a00a3690 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f10b4100690 0x7f10b419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.520 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10b40feb10 0x7f10b4195860 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 --2- 172.21.15.64:0/569465764 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f10b40ffd00 0x7f10b4195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.521 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 >> 172.21.15.64:0/569465764 conn(0x7f10b40fa130 msgr2=0x7f10b40faa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:12.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 shutdown_connections 2023-07-14T18:25:12.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.465+0000 7f10b8b09700 1 -- 172.21.15.64:0/569465764 wait complete. 2023-07-14T18:25:12.523 INFO:teuthology.orchestra.run.smithi064.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T18:25:12.540 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T18:25:12.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 -- 172.21.15.64:0/3006453257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900feb40 msgr2=0x7f79900fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 --2- 172.21.15.64:0/3006453257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900feb40 0x7f79900fef50 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f7980009990 tx=0x7f7980009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 -- 172.21.15.64:0/3006453257 shutdown_connections 2023-07-14T18:25:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 --2- 172.21.15.64:0/3006453257 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 --2- 172.21.15.64:0/3006453257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900feb40 0x7f79900fef50 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 --2- 172.21.15.64:0/3006453257 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900ffd30 0x7f7990100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 -- 172.21.15.64:0/3006453257 >> 172.21.15.64:0/3006453257 conn(0x7f79900fa160 msgr2=0x7f79900fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 -- 172.21.15.64:0/3006453257 shutdown_connections 2023-07-14T18:25:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.633+0000 7f799513d700 1 -- 172.21.15.64:0/3006453257 wait complete. 2023-07-14T18:25:12.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 Processor -- start 2023-07-14T18:25:12.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 -- start start 2023-07-14T18:25:12.690 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 0x7f79901014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990101a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79901021a0 con 0x7f79900ffd30 2023-07-14T18:25:12.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7990102310 con 0x7f79901006c0 2023-07-14T18:25:12.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f799513d700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799019f940 con 0x7f79900feb40 2023-07-14T18:25:12.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f798ed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f798e59c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 0x7f79901014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f798ed9d700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:56862/0 (socket says 172.21.15.64:56862) 2023-07-14T18:25:12.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f798ed9d700 1 -- 172.21.15.64:0/877421857 learned_addr learned my addr 172.21.15.64:0/877421857 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.634+0000 7f798f59e700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990101a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 -- 172.21.15.64:0/877421857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 msgr2=0x7f7990101a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990101a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 -- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 msgr2=0x7f79901014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:12.693 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 0x7f79901014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:12.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 -- 172.21.15.64:0/877421857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800095f0 con 0x7f79900feb40 2023-07-14T18:25:12.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798e59c700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 0x7f79901014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:25:12.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798f59e700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:25:12.694 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f798ed9d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f7980004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798001d070 con 0x7f79900feb40 2023-07-14T18:25:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f798000bd50 con 0x7f79900feb40 2023-07-14T18:25:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799019fbc0 con 0x7f79900feb40 2023-07-14T18:25:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7980022cd0 con 0x7f79900feb40 2023-07-14T18:25:12.695 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.635+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799019ffe0 con 0x7f79900feb40 2023-07-14T18:25:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.636+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7990106cc0 con 0x7f79900feb40 2023-07-14T18:25:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.637+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7980022470 con 0x7f79900feb40 2023-07-14T18:25:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.638+0000 7f7977fff700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 0x7f79780a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:12.696 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.638+0000 7f798e59c700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 0x7f79780a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:12.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.638+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9595+0+0 (secure 0 0 0) 0x7f79800ce930 con 0x7f79900feb40 2023-07-14T18:25:12.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.638+0000 7f798e59c700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 0x7f79780a3690 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7984011490 tx=0x7f7984009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:12.697 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.641+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7980097a00 con 0x7f79900feb40 2023-07-14T18:25:12.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1923975939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/1151154533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.813 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.813 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.814 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-mds[122598]: -- 172.21.15.105:0/3846237260 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559ce4cd91e0 con 0x559ce4cae800 2023-07-14T18:25:12.814 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: starting mds.e at 2023-07-14T18:25:12.815 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.815 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.816 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-mds[122946]: -- 172.21.15.105:0/3256034465 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5571000071e0 con 0x5570fffdcc00 2023-07-14T18:25:12.816 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: starting mds.h at 2023-07-14T18:25:12.817 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: -8031> 2023-07-14T18:23:30.278+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2023-07-14T18:25:12.817 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: -8029> 2023-07-14T18:23:30.278+0000 7fcec4537700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2023-07-14T18:25:12.818 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.818 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.818 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-mds[123288]: -- 172.21.15.105:0/745074417 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560165aa11e0 con 0x560165a76800 2023-07-14T18:25:12.819 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:25:12 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: starting mds.b at 2023-07-14T18:25:12.843 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.843 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.844 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: starting mds.j at 2023-07-14T18:25:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a[96696]: 2023-07-14T18:25:12.451+0000 7f420785a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:25:12.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1923975939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/1151154533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.845 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.846 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.846 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: starting mds.a at 2023-07-14T18:25:12.846 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.847 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.847 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: starting mds.d at 2023-07-14T18:25:12.847 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.847 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.848 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-mds[131102]: -- 172.21.15.64:0/3253175408 <== mon.0 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c5e73d91e0 con 0x55c5e73ae800 2023-07-14T18:25:12.848 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:25:12 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: starting mds.g at 2023-07-14T18:25:12.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1923975939' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/1151154533' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T18:25:12.934 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.935 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.935 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: starting mds.c at 2023-07-14T18:25:12.936 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.936 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.937 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-mds[124101]: -- 172.21.15.144:0/2882840588 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bed0fdf1e0 con 0x55bed0fb4c00 2023-07-14T18:25:12.937 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: starting mds.f at 2023-07-14T18:25:12.938 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: ignoring --setuser ceph since I am not root 2023-07-14T18:25:12.939 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: ignoring --setgroup ceph since I am not root 2023-07-14T18:25:12.939 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-mds[124781]: -- 172.21.15.144:0/772149876 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e5e0d251e0 con 0x55e5e0cfb000 2023-07-14T18:25:12.940 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:25:12 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: starting mds.i at 2023-07-14T18:25:12.947 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:12.896+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f799002cb10 con 0x7f79900feb40 2023-07-14T18:25:13.522 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.471+0000 7f7977fff700 1 -- 172.21.15.64:0/877421857 <== mon.1 v2:172.21.15.144:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) v1 ==== 104+0+0 (secure 0 0 0) 0x7f798009b020 con 0x7f79900feb40 2023-07-14T18:25:13.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 msgr2=0x7f79780a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:13.528 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 0x7f79780a3690 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7984011490 tx=0x7f7984009260 comp rx=0 tx=0).stop 2023-07-14T18:25:13.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 msgr2=0x7f7990102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:13.529 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f7980004730 comp rx=0 tx=0).stop 2023-07-14T18:25:13.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 shutdown_connections 2023-07-14T18:25:13.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f79900feb40 0x7f7990102e80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.530 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f79780a11e0 0x7f79780a3690 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79901006c0 0x7f7990101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.531 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 --2- 172.21.15.64:0/877421857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79900ffd30 0x7f79901014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 >> 172.21.15.64:0/877421857 conn(0x7f79900fa160 msgr2=0x7f79900fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:13.532 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 shutdown_connections 2023-07-14T18:25:13.533 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.476+0000 7f799513d700 1 -- 172.21.15.64:0/877421857 wait complete. 2023-07-14T18:25:13.543 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T18:25:13.677 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 -- 172.21.15.64:0/385980656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c105970 msgr2=0x7f862c107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:13.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 --2- 172.21.15.64:0/385980656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c105970 0x7f862c107d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8624009f90 tx=0x7f862400b330 comp rx=0 tx=0).stop 2023-07-14T18:25:13.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 -- 172.21.15.64:0/385980656 shutdown_connections 2023-07-14T18:25:13.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 --2- 172.21.15.64:0/385980656 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c105970 0x7f862c107d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 --2- 172.21.15.64:0/385980656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c100730 0x7f862c102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.678 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 --2- 172.21.15.64:0/385980656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 -- 172.21.15.64:0/385980656 >> 172.21.15.64:0/385980656 conn(0x7f862c0fa130 msgr2=0x7f862c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:13.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 -- 172.21.15.64:0/385980656 shutdown_connections 2023-07-14T18:25:13.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.625+0000 7f8632234700 1 -- 172.21.15.64:0/385980656 wait complete. 2023-07-14T18:25:13.679 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 Processor -- start 2023-07-14T18:25:13.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 -- start start 2023-07-14T18:25:13.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:13.681 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:13.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c105970 0x7f862c19cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:13.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c19d4e0 con 0x7f862c103050 2023-07-14T18:25:13.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c19d620 con 0x7f862c105970 2023-07-14T18:25:13.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f8632234700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c19d7b0 con 0x7f862c100730 2023-07-14T18:25:13.682 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:13.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:13.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862b7fe700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.144:3300/0 says I am v2:172.21.15.64:56886/0 (socket says 172.21.15.64:56886) 2023-07-14T18:25:13.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862affd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49232/0 (socket says 172.21.15.64:49232) 2023-07-14T18:25:13.683 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862b7fe700 1 -- 172.21.15.64:0/2442724824 learned_addr learned my addr 172.21.15.64:0/2442724824 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:13.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.626+0000 7f862bfff700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c105970 0x7f862c19cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:13.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 -- 172.21.15.64:0/2442724824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 msgr2=0x7f862c195960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:13.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.684 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 -- 172.21.15.64:0/2442724824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c105970 msgr2=0x7f862c19cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:13.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c105970 0x7f862c19cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:13.685 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 -- 172.21.15.64:0/2442724824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8624009c40 con 0x7f862c103050 2023-07-14T18:25:13.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862b7fe700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:25:13.686 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f862affd700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f862000e9c0 tx=0x7f862000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:13.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862000c9c0 con 0x7f862c103050 2023-07-14T18:25:13.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c19da90 con 0x7f862c103050 2023-07-14T18:25:13.687 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8620010430 con 0x7f862c103050 2023-07-14T18:25:13.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8620018680 con 0x7f862c103050 2023-07-14T18:25:13.688 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.627+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c19dfa0 con 0x7f862c103050 2023-07-14T18:25:13.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.628+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c10b270 con 0x7f862c103050 2023-07-14T18:25:13.689 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.629+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f86200105a0 con 0x7f862c103050 2023-07-14T18:25:13.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.630+0000 7f8628ff9700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 0x7f86140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:13.691 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.630+0000 7f862b7fe700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 0x7f86140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:13.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.630+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9595+0+0 (secure 0 0 0) 0x7f8620014070 con 0x7f862c103050 2023-07-14T18:25:13.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.630+0000 7f862b7fe700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 0x7f86140a3690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f861c011550 tx=0x7f861c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:13.692 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.633+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f86200963a0 con 0x7f862c103050 2023-07-14T18:25:13.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 895 B/s wr, 10 op/s 2023-07-14T18:25:13.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:25:13.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:25:13.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:25:13.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:25:13.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:25:13.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T18:25:13.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/877421857' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:13 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 895 B/s wr, 10 op/s 2023-07-14T18:25:13.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:25:13.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:25:13.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:25:13.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:25:13.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:25:13.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T18:25:13.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/877421857' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:13 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.890 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:13.838+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f862c19e6a0 con 0x7f862c103050 2023-07-14T18:25:13.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: pgmap v322: 129 pgs: 1 peering, 128 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 895 B/s wr, 10 op/s 2023-07-14T18:25:13.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T18:25:13.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T18:25:13.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T18:25:13.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T18:25:13.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/569465764' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T18:25:13.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: fsmap cephfs:0/5 2 up:standby, 5 failed 2023-07-14T18:25:13.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/877421857' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:25:13.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:13.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:13 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.573 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.521+0000 7f8628ff9700 1 -- 172.21.15.64:0/2442724824 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7f86200999c0 con 0x7f862c103050 2023-07-14T18:25:14.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 msgr2=0x7f86140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:14.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 0x7f86140a3690 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f861c011550 tx=0x7f861c009260 comp rx=0 tx=0).stop 2023-07-14T18:25:14.576 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 msgr2=0x7f862c195ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:14.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f862000e9c0 tx=0x7f862000ecd0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 shutdown_connections 2023-07-14T18:25:14.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f862c100730 0x7f862c195960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f86140a11e0 0x7f86140a3690 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.577 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f862c105970 0x7f862c19cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 --2- 172.21.15.64:0/2442724824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f862c103050 0x7f862c195ea0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 >> 172.21.15.64:0/2442724824 conn(0x7f862c0fa130 msgr2=0x7f862c101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:14.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 shutdown_connections 2023-07-14T18:25:14.578 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.524+0000 7f8632234700 1 -- 172.21.15.64:0/2442724824 wait complete. 2023-07-14T18:25:14.579 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_metadata' removed 2023-07-14T18:25:14.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T18:25:14.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 -- 172.21.15.64:0/411064886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 msgr2=0x7faa00102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:14.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 --2- 172.21.15.64:0/411064886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00102b10 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009990 tx=0x7fa9f0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 -- 172.21.15.64:0/411064886 shutdown_connections 2023-07-14T18:25:14.733 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 --2- 172.21.15.64:0/411064886 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 0x7faa00107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 --2- 172.21.15.64:0/411064886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00102b10 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 --2- 172.21.15.64:0/411064886 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 0x7faa00105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.680+0000 7faa067e4700 1 -- 172.21.15.64:0/411064886 >> 172.21.15.64:0/411064886 conn(0x7faa000fa130 msgr2=0x7faa000fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:14.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 -- 172.21.15.64:0/411064886 shutdown_connections 2023-07-14T18:25:14.734 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 -- 172.21.15.64:0/411064886 wait complete. 2023-07-14T18:25:14.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 Processor -- start 2023-07-14T18:25:14.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 -- start start 2023-07-14T18:25:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 0x7faa00195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.681+0000 7faa067e4700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 0x7faa0019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:14.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa067e4700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0019d400 con 0x7faa00100730 2023-07-14T18:25:14.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa067e4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0019d540 con 0x7faa00103050 2023-07-14T18:25:14.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa067e4700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0019d6d0 con 0x7faa00105970 2023-07-14T18:25:14.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:14.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49250/0 (socket says 172.21.15.64:49250) 2023-07-14T18:25:14.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 -- 172.21.15.64:0/823980677 learned_addr learned my addr 172.21.15.64:0/823980677 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:14.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa04d81700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 0x7faa0019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:14.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9ff7fe700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 0x7faa00195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:14.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 -- 172.21.15.64:0/823980677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 msgr2=0x7faa0019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:14.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 0x7faa0019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 -- 172.21.15.64:0/823980677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 msgr2=0x7faa00195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 0x7faa00195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 -- 172.21.15.64:0/823980677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f00095f0 con 0x7faa00100730 2023-07-14T18:25:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fffff700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00059d0 tx=0x7fa9f0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:14.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f001d070 con 0x7faa00100730 2023-07-14T18:25:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa9f000beb0 con 0x7faa00100730 2023-07-14T18:25:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa0019d950 con 0x7faa00100730 2023-07-14T18:25:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f000fdb0 con 0x7faa00100730 2023-07-14T18:25:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.682+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa0019de00 con 0x7faa00100730 2023-07-14T18:25:14.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.683+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa000fbca0 con 0x7faa00100730 2023-07-14T18:25:14.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.684+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa9f000f460 con 0x7faa00100730 2023-07-14T18:25:14.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.685+0000 7fa9fd7fa700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 0x7fa9e80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:14.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.685+0000 7fa9ff7fe700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 0x7fa9e80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:14.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.685+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9099+0+0 (secure 0 0 0) 0x7fa9f00ce330 con 0x7faa00100730 2023-07-14T18:25:14.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.685+0000 7fa9ff7fe700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 0x7fa9e80a3690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005950 tx=0x7fa9f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:14.743 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.688+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa9f00976f0 con 0x7faa00100730 2023-07-14T18:25:14.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:25:14.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:25:14.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6838/3538248422,v1:172.21.15.144:6839/3538248422] up:boot 2023-07-14T18:25:14.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6834/80288090,v1:172.21.15.144:6835/80288090] up:boot 2023-07-14T18:25:14.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6834/967318279,v1:172.21.15.105:6835/967318279] up:boot 2023-07-14T18:25:14.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.144:6836/3767082593,v1:172.21.15.144:6837/3767082593] up:boot 2023-07-14T18:25:14.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6839/15333327,v1:172.21.15.64:6840/15333327] up:boot 2023-07-14T18:25:14.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6838/911925876,v1:172.21.15.64:6841/911925876] up:boot 2023-07-14T18:25:14.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6834/608599570,v1:172.21.15.64:6835/608599570] up:boot 2023-07-14T18:25:14.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6836/4111942602,v1:172.21.15.105:6837/4111942602] up:boot 2023-07-14T18:25:14.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.105:6838/2905182987,v1:172.21.15.105:6839/2905182987] up:boot 2023-07-14T18:25:14.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: mds.? [v2:172.21.15.64:6836/3338294550,v1:172.21.15.64:6837/3338294550] up:boot 2023-07-14T18:25:14.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:25:14.813 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: fsmap 12 up:standby 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:25:14.814 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:25:14.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:25:14.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:25:14.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:25:14.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:25:14.815 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.816 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.816 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:14.816 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:25:14.816 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:25:14.816 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:14 smithi105 ceph-mon[98013]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:25:14.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:25:14.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6838/3538248422,v1:172.21.15.144:6839/3538248422] up:boot 2023-07-14T18:25:14.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6834/80288090,v1:172.21.15.144:6835/80288090] up:boot 2023-07-14T18:25:14.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6834/967318279,v1:172.21.15.105:6835/967318279] up:boot 2023-07-14T18:25:14.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.144:6836/3767082593,v1:172.21.15.144:6837/3767082593] up:boot 2023-07-14T18:25:14.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6839/15333327,v1:172.21.15.64:6840/15333327] up:boot 2023-07-14T18:25:14.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6838/911925876,v1:172.21.15.64:6841/911925876] up:boot 2023-07-14T18:25:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6834/608599570,v1:172.21.15.64:6835/608599570] up:boot 2023-07-14T18:25:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6836/4111942602,v1:172.21.15.105:6837/4111942602] up:boot 2023-07-14T18:25:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.105:6838/2905182987,v1:172.21.15.105:6839/2905182987] up:boot 2023-07-14T18:25:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: mds.? [v2:172.21.15.64:6836/3338294550,v1:172.21.15.64:6837/3338294550] up:boot 2023-07-14T18:25:14.844 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:25:14.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: fsmap 12 up:standby 2023-07-14T18:25:14.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:25:14.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:25:14.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:25:14.845 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:25:14.846 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:25:14.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:14.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:25:14.847 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:25:14.848 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:14 smithi064 ceph-mon[96720]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T18:25:14.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T18:25:14.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6838/3538248422,v1:172.21.15.144:6839/3538248422] up:boot 2023-07-14T18:25:14.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6834/80288090,v1:172.21.15.144:6835/80288090] up:boot 2023-07-14T18:25:14.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6834/967318279,v1:172.21.15.105:6835/967318279] up:boot 2023-07-14T18:25:14.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.144:6836/3767082593,v1:172.21.15.144:6837/3767082593] up:boot 2023-07-14T18:25:14.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6839/15333327,v1:172.21.15.64:6840/15333327] up:boot 2023-07-14T18:25:14.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6838/911925876,v1:172.21.15.64:6841/911925876] up:boot 2023-07-14T18:25:14.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6834/608599570,v1:172.21.15.64:6835/608599570] up:boot 2023-07-14T18:25:14.934 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6836/4111942602,v1:172.21.15.105:6837/4111942602] up:boot 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.105:6838/2905182987,v1:172.21.15.105:6839/2905182987] up:boot 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: mds.? [v2:172.21.15.64:6836/3338294550,v1:172.21.15.64:6837/3338294550] up:boot 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: fsmap 12 up:standby 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T18:25:14.935 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T18:25:14.936 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T18:25:14.936 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T18:25:14.936 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T18:25:14.937 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T18:25:14.937 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T18:25:14.937 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T18:25:14.937 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T18:25:14.938 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T18:25:14.938 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.938 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.938 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:14.938 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:25:14.939 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:25:14.939 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:14 smithi144 ceph-mon[98417]: from='mgr.14152 172.21.15.64:0/3519756555' entity='mgr.x' 2023-07-14T18:25:14.942 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:14.890+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7faa0002c8e0 con 0x7faa00100730 2023-07-14T18:25:15.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.524+0000 7fa9fd7fa700 1 -- 172.21.15.64:0/823980677 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa9f009ad10 con 0x7faa00100730 2023-07-14T18:25:15.579 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 msgr2=0x7fa9e80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 0x7fa9e80a3690 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4005950 tx=0x7fa9f400b410 comp rx=0 tx=0).stop 2023-07-14T18:25:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 msgr2=0x7faa00195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa9f00059d0 tx=0x7fa9f0004790 comp rx=0 tx=0).stop 2023-07-14T18:25:15.580 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 shutdown_connections 2023-07-14T18:25:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7faa00105970 0x7faa0019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7fa9e80a11e0 0x7fa9e80a3690 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faa00100730 0x7faa00195880 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 --2- 172.21.15.64:0/823980677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faa00103050 0x7faa00195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 >> 172.21.15.64:0/823980677 conn(0x7faa000fa130 msgr2=0x7faa00101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 shutdown_connections 2023-07-14T18:25:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.528+0000 7faa067e4700 1 -- 172.21.15.64:0/823980677 wait complete. 2023-07-14T18:25:15.582 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data' removed 2023-07-14T18:25:15.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-07-14T18:25:15.730 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 -- 172.21.15.64:0/3195555215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f10103010 msgr2=0x7f1f101053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:15.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195555215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f10103010 0x7f1f101053f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1ef8009990 tx=0x7f1ef8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 -- 172.21.15.64:0/3195555215 shutdown_connections 2023-07-14T18:25:15.731 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195555215 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f101006f0 0x7f1f10102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195555215 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10105930 0x7f1f10107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195555215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f10103010 0x7f1f101053f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 -- 172.21.15.64:0/3195555215 >> 172.21.15.64:0/3195555215 conn(0x7f1f100fa130 msgr2=0x7f1f100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:15.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 -- 172.21.15.64:0/3195555215 shutdown_connections 2023-07-14T18:25:15.732 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 -- 172.21.15.64:0/3195555215 wait complete. 2023-07-14T18:25:15.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.679+0000 7f1f16148700 1 Processor -- start 2023-07-14T18:25:15.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 -- start start 2023-07-14T18:25:15.735 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:15.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10103010 0x7f1f10195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:15.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 --2- >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f10105930 0x7f1f1019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:15.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019d460 con 0x7f1f101006f0 2023-07-14T18:25:15.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019d5a0 con 0x7f1f10103010 2023-07-14T18:25:15.736 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f16148700 1 -- --> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1019d730 con 0x7f1f10105930 2023-07-14T18:25:15.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:15.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49264/0 (socket says 172.21.15.64:49264) 2023-07-14T18:25:15.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 -- 172.21.15.64:0/3195485402 learned_addr learned my addr 172.21.15.64:0/3195485402 (peer_addr_for_me v2:172.21.15.64:0/0) 2023-07-14T18:25:15.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0effd700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10103010 0x7f1f10195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:15.737 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0ffff700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f10105930 0x7f1f1019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 -- 172.21.15.64:0/3195485402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f10105930 msgr2=0x7f1f1019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f10105930 0x7f1f1019ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 -- 172.21.15.64:0/3195485402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10103010 msgr2=0x7f1f10195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10103010 0x7f1f10195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:15.738 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.680+0000 7f1f0f7fe700 1 -- 172.21.15.64:0/3195485402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef80095f0 con 0x7f1f101006f0 2023-07-14T18:25:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f0f7fe700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000d950 tx=0x7f1f0000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f00009b40 con 0x7f1f101006f0 2023-07-14T18:25:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1f00009ca0 con 0x7f1f101006f0 2023-07-14T18:25:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1019da10 con 0x7f1f101006f0 2023-07-14T18:25:15.739 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0000f620 con 0x7f1f101006f0 2023-07-14T18:25:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.681+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1019df20 con 0x7f1f101006f0 2023-07-14T18:25:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.682+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f100fbca0 con 0x7f1f101006f0 2023-07-14T18:25:15.740 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.683+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1f00010460 con 0x7f1f101006f0 2023-07-14T18:25:15.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.683+0000 7f1f0cff9700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 0x7f1efc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:25:15.741 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.684+0000 7f1f0effd700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 0x7f1efc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:25:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.684+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8424+0+0 (secure 0 0 0) 0x7f1f000ccf70 con 0x7f1f101006f0 2023-07-14T18:25:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.684+0000 7f1f0effd700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 0x7f1efc0a3690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800b5c0 tx=0x7f1ef8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:25:15.742 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.688+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1f00096440 con 0x7f1f101006f0 2023-07-14T18:25:15.810 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 671 B/s wr, 12 op/s 2023-07-14T18:25:15.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-14T18:25:15.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: Cluster is now healthy 2023-07-14T18:25:15.811 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:15.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:25:15.812 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:15 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:15.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 671 B/s wr, 12 op/s 2023-07-14T18:25:15.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-14T18:25:15.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: Cluster is now healthy 2023-07-14T18:25:15.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:15.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:25:15.843 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:15 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:15.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: pgmap v324: 129 pgs: 129 active+clean; 650 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 671 B/s wr, 12 op/s 2023-07-14T18:25:15.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2023-07-14T18:25:15.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: Cluster is now healthy 2023-07-14T18:25:15.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/2442724824' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:15.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T18:25:15.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:15 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:15.941 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:15.889+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f1f10061250 con 0x7f1f101006f0 2023-07-14T18:25:16.575 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.523+0000 7f1f0cff9700 1 -- 172.21.15.64:0/3195485402 <== mon.0 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v93) v1 ==== 178+0+0 (secure 0 0 0) 0x7f1f000200d0 con 0x7f1f101006f0 2023-07-14T18:25:16.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 msgr2=0x7f1efc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:16.581 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 0x7f1efc0a3690 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800b5c0 tx=0x7f1ef8005fb0 comp rx=0 tx=0).stop 2023-07-14T18:25:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 msgr2=0x7f1f101958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:25:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1f0000d950 tx=0x7f1f0000dd10 comp rx=0 tx=0).stop 2023-07-14T18:25:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 shutdown_connections 2023-07-14T18:25:16.582 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] conn(0x7f1f10105930 0x7f1f1019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:6800/387190897,v1:172.21.15.64:6801/387190897] conn(0x7f1efc0a11e0 0x7f1efc0a3690 secure :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1ef800b5c0 tx=0x7f1ef8005fb0 comp rx=0 tx=0).stop 2023-07-14T18:25:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f101006f0 0x7f1f101958e0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 --2- 172.21.15.64:0/3195485402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f10103010 0x7f1f10195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:25:16.583 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.529+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 >> 172.21.15.64:0/3195485402 conn(0x7f1f100fa130 msgr2=0x7f1f10101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:25:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.530+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 shutdown_connections 2023-07-14T18:25:16.584 INFO:teuthology.orchestra.run.smithi064.stderr:2023-07-14T18:25:16.530+0000 7f1f16148700 1 -- 172.21.15.64:0/3195485402 wait complete. 2023-07-14T18:25:16.585 INFO:teuthology.orchestra.run.smithi064.stderr:pool 'cephfs_data_ec' removed 2023-07-14T18:25:16.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:25:16.631 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:25:16.667 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:25:16.716 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T18:25:16.717 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:25:16.735 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:25:16.753 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:25:16.772 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T18:25:16.773 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T18:25:16.773 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a 2023-07-14T18:25:16.841 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:16 smithi064 ceph-mon[96720]: pgmap v326: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:25:16.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:16 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:16.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:16 smithi064 ceph-mon[96720]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:25:16.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:16 smithi064 ceph-mon[96720]: from='client.? 172.21.15.64:0/3195485402' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:16.842 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:16 smithi064 systemd[1]: Stopping Ceph mon.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:16.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:16 smithi144 ceph-mon[98417]: pgmap v326: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:25:16.932 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:16 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:16.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:16 smithi144 ceph-mon[98417]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:25:16.933 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:16 smithi144 ceph-mon[98417]: from='client.? 172.21.15.64:0/3195485402' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:17.060 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:16 smithi105 ceph-mon[98013]: pgmap v326: 97 pgs: 97 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T18:25:17.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:16 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/823980677' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T18:25:17.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:16 smithi105 ceph-mon[98013]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T18:25:17.061 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:16 smithi105 ceph-mon[98013]: from='client.? 172.21.15.64:0/3195485402' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T18:25:17.431 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:17 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a[96696]: 2023-07-14T18:25:17.077+0000 7f420d065700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:17.432 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:17 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a[96696]: 2023-07-14T18:25:17.077+0000 7f420d065700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T18:25:17.682 INFO:journalctl@ceph.mon.a.smithi064.stdout:Jul 14 18:25:17 smithi064 bash[149987]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-a 2023-07-14T18:25:17.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T18:25:17.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:17.934 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T18:25:17.935 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T18:25:17.936 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.b 2023-07-14T18:25:18.310 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:17 smithi105 systemd[1]: Stopping Ceph mon.b for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:18.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:18 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-b[97989]: 2023-07-14T18:25:18.365+0000 7fb11c2d7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:18.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:18 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-b[97989]: 2023-07-14T18:25:18.365+0000 7fb11c2d7700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T18:25:19.054 INFO:journalctl@ceph.mon.b.smithi105.stdout:Jul 14 18:25:18 smithi105 bash[125397]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-b 2023-07-14T18:25:19.061 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T18:25:19.111 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:19.111 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T18:25:19.112 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T18:25:19.112 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.c 2023-07-14T18:25:19.431 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:19 smithi144 systemd[1]: Stopping Ceph mon.c for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:19.788 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:19 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-c[98393]: 2023-07-14T18:25:19.428+0000 7f6fc7fdd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:19.789 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:19 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-c[98393]: 2023-07-14T18:25:19.428+0000 7f6fc7fdd700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T18:25:20.142 INFO:journalctl@ceph.mon.c.smithi144.stdout:Jul 14 18:25:19 smithi144 bash[126854]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mon-c 2023-07-14T18:25:20.149 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T18:25:20.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:20.197 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T18:25:20.198 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T18:25:20.198 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T18:25:20.562 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:25:20 smithi064 systemd[1]: Stopping Ceph mgr.x for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:20.834 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:25:20 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:25:20.510+0000 7f9554a78700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:20.835 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:25:20 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x[96968]: 2023-07-14T18:25:20.510+0000 7f9554a78700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-14T18:25:21.091 INFO:journalctl@ceph.mgr.x.smithi064.stdout:Jul 14 18:25:20 smithi064 bash[150128]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T18:25:21.107 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T18:25:21.145 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:21.146 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T18:25:21.146 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T18:25:21.146 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T18:25:21.559 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:25:21 smithi105 systemd[1]: Stopping Ceph mgr.y for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:21.853 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:25:21 smithi105 bash[125548]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T18:25:22.034 INFO:journalctl@ceph.mgr.y.smithi105.stdout:Jul 14 18:25:21 smithi105 systemd[1]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T18:25:22.037 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T18:25:22.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:22.075 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T18:25:22.076 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T18:25:22.076 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T18:25:22.431 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:25:22 smithi144 systemd[1]: Stopping Ceph mgr.z for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:22.783 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:25:22 smithi144 bash[127012]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T18:25:22.962 INFO:journalctl@ceph.mgr.z.smithi144.stdout:Jul 14 18:25:22 smithi144 systemd[1]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T18:25:22.966 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T18:25:23.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:23.002 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T18:25:23.003 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T18:25:23.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.0 2023-07-14T18:25:23.340 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:25:22 smithi064 systemd[1]: Stopping Ceph osd.0 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:23.841 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:25:23 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:25:23.318+0000 7ff7ed357700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:23.841 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:25:23 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:25:23.318+0000 7ff7ed357700 -1 osd.0 93 *** Got signal Terminated *** 2023-07-14T18:25:23.841 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:25:23 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0[109812]: 2023-07-14T18:25:23.318+0000 7ff7ed357700 -1 osd.0 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:25:29.145 INFO:journalctl@ceph.osd.0.smithi064.stdout:Jul 14 18:25:29 smithi064 bash[150269]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-0 2023-07-14T18:25:30.421 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T18:25:30.463 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:30.464 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T18:25:30.464 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T18:25:30.464 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.1 2023-07-14T18:25:30.809 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:25:30 smithi105 systemd[1]: Stopping Ceph osd.1 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:31.309 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:25:30 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:25:30.765+0000 7f839790d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:31.310 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:25:30 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:25:30.765+0000 7f839790d700 -1 osd.1 93 *** Got signal Terminated *** 2023-07-14T18:25:31.310 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:25:30 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1[102966]: 2023-07-14T18:25:30.765+0000 7f839790d700 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:25:36.808 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jul 14 18:25:36 smithi105 bash[125692]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-1 2023-07-14T18:25:37.868 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T18:25:37.919 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:37.920 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T18:25:37.920 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T18:25:37.920 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.2 2023-07-14T18:25:38.291 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:25:37 smithi144 systemd[1]: Stopping Ceph osd.2 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:38.680 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:25:38 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:25:38.241+0000 7f2224c56700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:38.680 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:25:38 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:25:38.241+0000 7f2224c56700 -1 osd.2 93 *** Got signal Terminated *** 2023-07-14T18:25:38.680 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:25:38 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2[104250]: 2023-07-14T18:25:38.241+0000 7f2224c56700 -1 osd.2 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:25:44.179 INFO:journalctl@ceph.osd.2.smithi144.stdout:Jul 14 18:25:43 smithi144 bash[127156]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-2 2023-07-14T18:25:45.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T18:25:45.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:45.422 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T18:25:45.423 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T18:25:45.423 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.3 2023-07-14T18:25:45.790 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:25:45 smithi064 systemd[1]: Stopping Ceph osd.3 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:46.089 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:25:45 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:25:45.741+0000 7f0f7e9eb700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:46.089 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:25:45 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:25:45.741+0000 7f0f7e9eb700 -1 osd.3 93 *** Got signal Terminated *** 2023-07-14T18:25:46.089 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:25:45 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3[115113]: 2023-07-14T18:25:45.741+0000 7f0f7e9eb700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:25:51.575 INFO:journalctl@ceph.osd.3.smithi064.stdout:Jul 14 18:25:51 smithi064 bash[150543]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-3 2023-07-14T18:25:52.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T18:25:52.890 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:25:52.891 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T18:25:52.891 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T18:25:52.891 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.4 2023-07-14T18:25:53.243 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:25:52 smithi105 systemd[1]: Stopping Ceph osd.4 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:25:53.556 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:25:53 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:25:53.194+0000 7f84e69c5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:25:53.557 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:25:53 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:25:53.194+0000 7f84e69c5700 -1 osd.4 93 *** Got signal Terminated *** 2023-07-14T18:25:53.557 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:25:53 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4[108378]: 2023-07-14T18:25:53.194+0000 7f84e69c5700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:25:59.306 INFO:journalctl@ceph.osd.4.smithi105.stdout:Jul 14 18:25:59 smithi105 bash[125972]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-4 2023-07-14T18:26:00.564 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T18:26:00.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:00.605 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T18:26:00.605 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T18:26:00.606 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.5 2023-07-14T18:26:00.928 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:26:00 smithi144 systemd[1]: Stopping Ceph osd.5 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:01.428 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:26:00 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:26:00.930+0000 7fe1fc257700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:01.428 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:26:00 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:26:00.930+0000 7fe1fc257700 -1 osd.5 93 *** Got signal Terminated *** 2023-07-14T18:26:01.428 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:26:00 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5[109718]: 2023-07-14T18:26:00.930+0000 7fe1fc257700 -1 osd.5 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:06.745 INFO:journalctl@ceph.osd.5.smithi144.stdout:Jul 14 18:26:06 smithi144 bash[127434]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-5 2023-07-14T18:26:08.210 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T18:26:08.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:08.251 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T18:26:08.251 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T18:26:08.252 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.6 2023-07-14T18:26:08.587 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:26:08 smithi064 systemd[1]: Stopping Ceph osd.6 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:09.087 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:26:08 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:26:08.570+0000 7fa1a10fc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:09.087 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:26:08 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:26:08.570+0000 7fa1a10fc700 -1 osd.6 93 *** Got signal Terminated *** 2023-07-14T18:26:09.087 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:26:08 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6[120653]: 2023-07-14T18:26:08.570+0000 7fa1a10fc700 -1 osd.6 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:14.586 INFO:journalctl@ceph.osd.6.smithi064.stdout:Jul 14 18:26:14 smithi064 bash[150799]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-6 2023-07-14T18:26:15.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T18:26:15.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:15.810 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T18:26:15.811 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T18:26:15.811 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.7 2023-07-14T18:26:16.172 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:26:15 smithi105 systemd[1]: Stopping Ceph osd.7 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:16.555 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:26:16 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:26:16.126+0000 7ff341554700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:16.555 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:26:16 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:26:16.126+0000 7ff341554700 -1 osd.7 93 *** Got signal Terminated *** 2023-07-14T18:26:16.555 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:26:16 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7[113810]: 2023-07-14T18:26:16.126+0000 7ff341554700 -1 osd.7 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:22.055 INFO:journalctl@ceph.osd.7.smithi105.stdout:Jul 14 18:26:21 smithi105 bash[126242]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-7 2023-07-14T18:26:23.108 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T18:26:23.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:23.149 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T18:26:23.150 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T18:26:23.150 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.8 2023-07-14T18:26:23.513 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:26:23 smithi144 systemd[1]: Stopping Ceph osd.8 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:23.926 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:26:23 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:26:23.467+0000 7f68886a1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:23.927 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:26:23 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:26:23.467+0000 7f68886a1700 -1 osd.8 93 *** Got signal Terminated *** 2023-07-14T18:26:23.927 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:26:23 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8[115233]: 2023-07-14T18:26:23.467+0000 7f68886a1700 -1 osd.8 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:29.426 INFO:journalctl@ceph.osd.8.smithi144.stdout:Jul 14 18:26:29 smithi144 bash[127697]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-8 2023-07-14T18:26:29.426 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:29 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:29.143+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:30.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:30 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:30.155+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:30.685 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T18:26:30.726 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:30.726 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T18:26:30.726 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T18:26:30.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.9 2023-07-14T18:26:31.085 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:30 smithi064 systemd[1]: Stopping Ceph osd.9 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:31.585 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:31.052+0000 7fe7c41de700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:31.585 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:31.052+0000 7fe7c41de700 -1 osd.9 93 *** Got signal Terminated *** 2023-07-14T18:26:31.586 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:31 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:31.052+0000 7fe7c41de700 -1 osd.9 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:31.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:31 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:31.169+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:32.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:31 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:31.901+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:32.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:32 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:32.130+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:32.834 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:32 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:32.522+0000 7fe7be7d1700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:26.109720+0000 front 2023-07-14T18:25:26.109868+0000 (oldest deadline 2023-07-14T18:26:31.910725+0000) 2023-07-14T18:26:33.304 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:32 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:32.854+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:33.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:33 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:33.159+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:34.084 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:33 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:33.552+0000 7fe7be7d1700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:26.109720+0000 front 2023-07-14T18:25:26.109868+0000 (oldest deadline 2023-07-14T18:26:31.910725+0000) 2023-07-14T18:26:34.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:33 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:33.837+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:34.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:34 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:34.186+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:34.834 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:34 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:34.536+0000 7fe7be7d1700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:26.109720+0000 front 2023-07-14T18:25:26.109868+0000 (oldest deadline 2023-07-14T18:26:31.910725+0000) 2023-07-14T18:26:35.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:34 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:34.820+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:35.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:35 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:35.162+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:35.835 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:35 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9[126193]: 2023-07-14T18:26:35.530+0000 7fe7be7d1700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:26.109720+0000 front 2023-07-14T18:25:26.109868+0000 (oldest deadline 2023-07-14T18:26:31.910725+0000) 2023-07-14T18:26:36.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:35 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:35.820+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:36.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:36 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:36.131+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:37.085 INFO:journalctl@ceph.osd.9.smithi064.stdout:Jul 14 18:26:36 smithi064 bash[151062]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-9 2023-07-14T18:26:37.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:36 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:36.810+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:37.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:37 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:37.143+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:37.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:37 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:37.143+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:38.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T18:26:38.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:37 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:37.814+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:38.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:38.337 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T18:26:38.338 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T18:26:38.339 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.10 2023-07-14T18:26:38.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:38 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:38.107+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:38.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:38 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:38.107+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:38.692 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:38 smithi105 systemd[1]: Stopping Ceph osd.10 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:38.692 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:38 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:38.648+0000 7f8aafc38700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:39.053 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:38 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:38.648+0000 7f8aafc38700 -1 osd.10 93 *** Got signal Terminated *** 2023-07-14T18:26:39.053 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:38 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:38.648+0000 7f8aafc38700 -1 osd.10 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:39.054 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:38 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:38.849+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:39.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:39 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:39.115+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:39.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:39 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:39.115+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:40.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:39 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:39.893+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:40.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:40 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:40.067+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:40.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:40 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:40.067+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:41.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:40 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:40.861+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:41.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:40 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:40.861+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:34.000305+0000 front 2023-07-14T18:25:34.000325+0000 (oldest deadline 2023-07-14T18:26:39.900452+0000) 2023-07-14T18:26:41.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:41 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:41.109+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:41.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:41 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:41.109+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:42.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:41 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:41.828+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:42.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:41 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:41.828+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:34.000305+0000 front 2023-07-14T18:25:34.000325+0000 (oldest deadline 2023-07-14T18:26:39.900452+0000) 2023-07-14T18:26:42.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:42 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:42.119+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:42.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:42 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:42.119+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:43.303 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:42 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:42.826+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:25.899060+0000 front 2023-07-14T18:25:25.898956+0000 (oldest deadline 2023-07-14T18:26:31.198939+0000) 2023-07-14T18:26:43.304 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:42 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10[119373]: 2023-07-14T18:26:42.826+0000 7f8aa9229700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:34.000305+0000 front 2023-07-14T18:25:34.000325+0000 (oldest deadline 2023-07-14T18:26:39.900452+0000) 2023-07-14T18:26:43.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:43 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:43.123+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:43.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:43 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:43.123+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:44.553 INFO:journalctl@ceph.osd.10.smithi105.stdout:Jul 14 18:26:44 smithi105 bash[126506]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-10 2023-07-14T18:26:44.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:44 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:44.148+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:44.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:44 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:44.148+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:45.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:45 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:45.169+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:45.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:45 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:45.169+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:45.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:45 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:45.169+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:45.738 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T18:26:45.778 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:45.779 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T18:26:45.780 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T18:26:45.780 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.11 2023-07-14T18:26:46.149 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:45 smithi144 systemd[1]: Stopping Ceph osd.11 for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:46.149 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.104+0000 7f01feacd700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:46.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.104+0000 7f01feacd700 -1 osd.11 93 *** Got signal Terminated *** 2023-07-14T18:26:46.424 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.104+0000 7f01feacd700 -1 osd.11 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T18:26:46.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.150+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:46.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.150+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:46.425 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:46 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:46.150+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:47.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:47 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:47.153+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:47.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:47 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:47.153+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:47.675 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:47 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:47.153+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:48.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:48 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:48.170+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:48.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:48 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:48.170+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:48.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:48 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:48.170+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:49.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:49 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:49.202+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:49.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:49 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:49.202+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:49.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:49 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:49.202+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:50.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:50 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:50.246+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.64:6806 osd.0 since back 2023-07-14T18:25:24.892045+0000 front 2023-07-14T18:25:24.891986+0000 (oldest deadline 2023-07-14T18:26:28.992170+0000) 2023-07-14T18:26:50.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:50 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:50.246+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.105:6804 osd.1 since back 2023-07-14T18:25:35.393927+0000 front 2023-07-14T18:25:35.393909+0000 (oldest deadline 2023-07-14T18:26:37.093849+0000) 2023-07-14T18:26:50.674 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:50 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11[120822]: 2023-07-14T18:26:50.246+0000 7f01fb0c4700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.144:6804 osd.2 since back 2023-07-14T18:25:42.895441+0000 front 2023-07-14T18:25:42.895480+0000 (oldest deadline 2023-07-14T18:26:44.595561+0000) 2023-07-14T18:26:51.818 INFO:journalctl@ceph.osd.11.smithi144.stdout:Jul 14 18:26:51 smithi144 bash[127968]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-osd-11 2023-07-14T18:26:53.181 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T18:26:53.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:53.223 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T18:26:53.224 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T18:26:53.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.a 2023-07-14T18:26:53.583 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:26:53 smithi064 systemd[1]: Stopping Ceph mds.a for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:54.083 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:26:53 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: 2023-07-14T18:26:53.547+0000 7f274b1a6700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:54.083 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:26:53 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a[131463]: 2023-07-14T18:26:53.547+0000 7f274b1a6700 -1 mds.a *** got signal Terminated *** 2023-07-14T18:26:57.083 INFO:journalctl@ceph.mds.a.smithi064.stdout:Jul 14 18:26:56 smithi064 bash[151342]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-a 2023-07-14T18:26:57.111 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T18:26:57.149 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:26:57.149 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T18:26:57.150 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T18:26:57.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.d 2023-07-14T18:26:57.531 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:26:57 smithi064 systemd[1]: Stopping Ceph mds.d for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:26:57.832 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:26:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: 2023-07-14T18:26:57.488+0000 7f567b7b8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:26:57.833 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:26:57 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d[130466]: 2023-07-14T18:26:57.488+0000 7f567b7b8700 -1 mds.d *** got signal Terminated *** 2023-07-14T18:27:01.195 INFO:journalctl@ceph.mds.d.smithi064.stdout:Jul 14 18:27:00 smithi064 bash[151488]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-d 2023-07-14T18:27:01.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T18:27:01.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:01.241 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T18:27:01.241 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T18:27:01.242 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.g 2023-07-14T18:27:01.582 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:27:01 smithi064 systemd[1]: Stopping Ceph mds.g for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:02.083 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:27:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: 2023-07-14T18:27:01.587+0000 7f3f54096700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:02.083 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:27:01 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g[131078]: 2023-07-14T18:27:01.587+0000 7f3f54096700 -1 mds.g *** got signal Terminated *** 2023-07-14T18:27:04.889 INFO:journalctl@ceph.mds.g.smithi064.stdout:Jul 14 18:27:04 smithi064 bash[151629]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-g 2023-07-14T18:27:05.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T18:27:05.122 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:05.123 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T18:27:05.123 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T18:27:05.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.j 2023-07-14T18:27:05.332 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:27:05 smithi064 systemd[1]: Stopping Ceph mds.j for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:05.834 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:27:05 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: 2023-07-14T18:27:05.469+0000 7f1ec8e90700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:05.834 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:27:05 smithi064 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j[130106]: 2023-07-14T18:27:05.469+0000 7f1ec8e90700 -1 mds.j *** got signal Terminated *** 2023-07-14T18:27:08.876 INFO:journalctl@ceph.mds.j.smithi064.stdout:Jul 14 18:27:08 smithi064 bash[151768]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-j 2023-07-14T18:27:09.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T18:27:09.113 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:09.113 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T18:27:09.114 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T18:27:09.114 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.b 2023-07-14T18:27:09.469 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:27:09 smithi105 systemd[1]: Stopping Ceph mds.b for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:09.801 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:27:09 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:27:09.427+0000 7f7bf8276700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:09.801 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:27:09 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b[123264]: 2023-07-14T18:27:09.427+0000 7f7bf8276700 -1 mds.b *** got signal Terminated *** 2023-07-14T18:27:13.047 INFO:journalctl@ceph.mds.b.smithi105.stdout:Jul 14 18:27:12 smithi105 bash[126772]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-b 2023-07-14T18:27:13.053 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T18:27:13.089 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:13.089 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T18:27:13.090 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T18:27:13.090 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.e 2023-07-14T18:27:13.300 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:27:13 smithi105 systemd[1]: Stopping Ceph mds.e for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:13.800 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:27:13 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: 2023-07-14T18:27:13.418+0000 7f9b8c1a5700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:13.801 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:27:13 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e[122574]: 2023-07-14T18:27:13.418+0000 7f9b8c1a5700 -1 mds.e *** got signal Terminated *** 2023-07-14T18:27:17.033 INFO:journalctl@ceph.mds.e.smithi105.stdout:Jul 14 18:27:16 smithi105 bash[126921]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-e 2023-07-14T18:27:17.039 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T18:27:17.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:17.074 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T18:27:17.075 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T18:27:17.075 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.h 2023-07-14T18:27:17.300 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:27:17 smithi105 systemd[1]: Stopping Ceph mds.h for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:17.800 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:27:17 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: 2023-07-14T18:27:17.397+0000 7f64f9b70700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:17.801 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:27:17 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h[122921]: 2023-07-14T18:27:17.397+0000 7f64f9b70700 -1 mds.h *** got signal Terminated *** 2023-07-14T18:27:20.996 INFO:journalctl@ceph.mds.h.smithi105.stdout:Jul 14 18:27:20 smithi105 bash[127072]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-h 2023-07-14T18:27:21.003 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T18:27:21.036 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:21.036 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T18:27:21.037 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T18:27:21.037 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.k 2023-07-14T18:27:21.300 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:27:21 smithi105 systemd[1]: Stopping Ceph mds.k for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:21.800 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:27:21 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-k[122235]: 2023-07-14T18:27:21.366+0000 7f6c4f2a8700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:21.800 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:27:21 smithi105 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-k[122235]: 2023-07-14T18:27:21.366+0000 7f6c4f2a8700 -1 mds.k *** got signal Terminated *** 2023-07-14T18:27:25.575 INFO:journalctl@ceph.mds.k.smithi105.stdout:Jul 14 18:27:25 smithi105 bash[127211]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-k 2023-07-14T18:27:25.581 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T18:27:25.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:25.616 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T18:27:25.616 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T18:27:25.617 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.c 2023-07-14T18:27:25.921 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:27:25 smithi144 systemd[1]: Stopping Ceph mds.c for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:26.421 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:27:25 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: 2023-07-14T18:27:25.947+0000 7faab8157700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:26.422 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:27:25 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c[124416]: 2023-07-14T18:27:25.947+0000 7faab8157700 -1 mds.c *** got signal Terminated *** 2023-07-14T18:27:29.171 INFO:journalctl@ceph.mds.c.smithi144.stdout:Jul 14 18:27:28 smithi144 bash[128239]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-c 2023-07-14T18:27:29.185 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T18:27:29.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:29.223 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T18:27:29.223 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T18:27:29.223 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.f 2023-07-14T18:27:29.613 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:27:29 smithi144 systemd[1]: Stopping Ceph mds.f for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:29.921 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:27:29 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: 2023-07-14T18:27:29.571+0000 7f2b99df3700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:29.921 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:27:29 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f[124077]: 2023-07-14T18:27:29.571+0000 7f2b99df3700 -1 mds.f *** got signal Terminated *** 2023-07-14T18:27:32.870 INFO:journalctl@ceph.mds.f.smithi144.stdout:Jul 14 18:27:32 smithi144 bash[128387]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-f 2023-07-14T18:27:33.056 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T18:27:33.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:33.091 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T18:27:33.092 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T18:27:33.093 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.i 2023-07-14T18:27:33.420 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:27:33 smithi144 systemd[1]: Stopping Ceph mds.i for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:33.921 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:27:33 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: 2023-07-14T18:27:33.437+0000 7f9892d6c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:33.922 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:27:33 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i[124757]: 2023-07-14T18:27:33.437+0000 7f9892d6c700 -1 mds.i *** got signal Terminated *** 2023-07-14T18:27:37.170 INFO:journalctl@ceph.mds.i.smithi144.stdout:Jul 14 18:27:36 smithi144 bash[128530]: ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-i 2023-07-14T18:27:37.220 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T18:27:37.253 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:37.254 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T18:27:37.254 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T18:27:37.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.l 2023-07-14T18:27:37.635 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:27:37 smithi144 systemd[1]: Stopping Ceph mds.l for 5441c8fe-2272-11ee-9b33-001a4aab830c... 2023-07-14T18:27:37.920 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:27:37 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-l[123734]: 2023-07-14T18:27:37.594+0000 7fefa90fa700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:27:37.921 INFO:journalctl@ceph.mds.l.smithi144.stdout:Jul 14 18:27:37 smithi144 ceph-5441c8fe-2272-11ee-9b33-001a4aab830c-mds-l[123734]: 2023-07-14T18:27:37.594+0000 7fefa90fa700 -1 mds.l *** got signal Terminated *** 2023-07-14T18:27:39.486 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-5441c8fe-2272-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T18:27:39.520 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T18:27:39.521 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T18:27:39.521 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:27:39.829 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:41.469 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:27:41.765 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:43.316 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T18:27:43.621 INFO:teuthology.orchestra.run.smithi144.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:45.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:27:45.219 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:27:45.255 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:27:45.285 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T18:27:45.288 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi064/crash 2023-07-14T18:27:45.289 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:27:45.335 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:27:45.336 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:27:45.340 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi105/crash 2023-07-14T18:27:45.342 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:27:45.383 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:27:45.384 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:27:45.385 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi144/crash 2023-07-14T18:27:45.386 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T18:27:45.422 INFO:teuthology.orchestra.run.smithi144.stderr:tar: /var/lib/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T18:27:45.423 INFO:teuthology.orchestra.run.smithi144.stderr:tar: Error is not recoverable: exiting now 2023-07-14T18:27:45.424 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T18:27:45.424 DEBUG:teuthology.orchestra.run.smithi064:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2023-07-14T18:27:45.461 INFO:teuthology.orchestra.run.smithi064.stderr:grep: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T18:27:45.464 INFO:tasks.cephadm:Compressing logs... 2023-07-14T18:27:45.464 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:27:45.507 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:27:45.510 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T18:27:45.536 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T18:27:45.536 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2023-07-14T18:27:45.536 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T18:27:45.538 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T18:27:45.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T18:27:45.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T18:27:45.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T18:27:45.539 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T18:27:45.540 INFO:teuthology.orchestra.run.smithi144.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T18:27:45.540 INFO:teuthology.orchestra.run.smithi144.stderr:: No such file or directory 2023-07-14T18:27:45.540 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T18:27:45.540 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T18:27:45.541 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T18:27:45.541 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.2.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T18:27:45.541 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T18:27:45.542 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.8.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.f.log: 91.7% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T18:27:45.542 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T18:27:45.543 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T18:27:45.543 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-07-14T18:27:45.544 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T18:27:45.544 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T18:27:45.544 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T18:27:45.545 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T18:27:45.545 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.h.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T18:27:45.545 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T18:27:45.546 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:27:45.546 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.k.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log: 94.8% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T18:27:45.546 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.c.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T18:27:45.547 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:27:45.547 INFO:teuthology.orchestra.run.smithi105.stderr: 91.5% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:27:45.548 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T18:27:45.548 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.d.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T18:27:45.549 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T18:27:45.551 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.7.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T18:27:45.553 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.b.log: 91.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T18:27:45.553 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T18:27:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T18:27:45.554 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.9.log: 92.7% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T18:27:45.554 INFO:teuthology.orchestra.run.smithi064.stderr: 92.7% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T18:27:45.554 INFO:teuthology.orchestra.run.smithi064.stderr: 92.6% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T18:27:45.555 INFO:teuthology.orchestra.run.smithi144.stderr: 91.5% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T18:27:45.558 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T18:27:45.560 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:27:45.560 INFO:teuthology.orchestra.run.smithi105.stderr: 92.6% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T18:27:45.560 INFO:teuthology.orchestra.run.smithi105.stderr: 91.5% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T18:27:45.561 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:27:45.563 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log: /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T18:27:45.564 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T18:27:45.564 INFO:teuthology.orchestra.run.smithi144.stderr: 91.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T18:27:45.565 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.l.log: 91.0% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T18:27:45.565 INFO:teuthology.orchestra.run.smithi144.stderr: 91.5% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:27:45.565 INFO:teuthology.orchestra.run.smithi144.stderr: 94.8% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T18:27:45.566 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:27:45.566 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/cephadm.log: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:27:45.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T18:27:45.574 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T18:27:45.574 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T18:27:45.575 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-14T18:27:45.577 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.log: 92.7% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T18:27:45.578 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2023-07-14T18:27:45.578 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.0.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-14T18:27:45.578 INFO:teuthology.orchestra.run.smithi064.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2023-07-14T18:27:45.614 INFO:teuthology.orchestra.run.smithi105.stderr: 92.7% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T18:27:45.695 INFO:teuthology.orchestra.run.smithi064.stderr: 89.0% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T18:27:45.717 INFO:teuthology.orchestra.run.smithi144.stderr: 92.8% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T18:27:45.752 INFO:teuthology.orchestra.run.smithi144.stderr: 91.9% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T18:27:45.758 INFO:teuthology.orchestra.run.smithi105.stderr: 91.9% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T18:27:45.766 INFO:teuthology.orchestra.run.smithi144.stderr: 92.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T18:27:45.804 INFO:teuthology.orchestra.run.smithi105.stderr: 92.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T18:27:45.814 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T18:27:45.822 INFO:teuthology.orchestra.run.smithi105.stderr: 92.4% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T18:27:45.839 INFO:teuthology.orchestra.run.smithi064.stderr: 92.2% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T18:27:45.844 INFO:teuthology.orchestra.run.smithi144.stderr: 92.2% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T18:27:45.858 INFO:teuthology.orchestra.run.smithi105.stderr: 92.4% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T18:27:45.867 INFO:teuthology.orchestra.run.smithi064.stderr: 92.4% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T18:27:45.869 INFO:teuthology.orchestra.run.smithi064.stderr: 92.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T18:27:45.879 INFO:teuthology.orchestra.run.smithi144.stderr: 92.3% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T18:27:45.881 INFO:teuthology.orchestra.run.smithi144.stderr: 2023-07-14T18:27:45.881 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m0.356s 2023-07-14T18:27:45.882 INFO:teuthology.orchestra.run.smithi144.stderr:user 0m1.306s 2023-07-14T18:27:45.882 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m0.060s 2023-07-14T18:27:45.896 INFO:teuthology.orchestra.run.smithi105.stderr: 92.1% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T18:27:45.898 INFO:teuthology.orchestra.run.smithi105.stderr: 2023-07-14T18:27:45.898 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m0.373s 2023-07-14T18:27:45.898 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m1.490s 2023-07-14T18:27:45.899 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.074s 2023-07-14T18:27:45.983 INFO:teuthology.orchestra.run.smithi064.stderr: 91.0% -- replaced with /var/log/ceph/5441c8fe-2272-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T18:27:45.985 INFO:teuthology.orchestra.run.smithi064.stderr: 2023-07-14T18:27:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.463s 2023-07-14T18:27:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m1.839s 2023-07-14T18:27:45.986 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.083s 2023-07-14T18:27:45.986 INFO:tasks.cephadm:Archiving logs... 2023-07-14T18:27:45.987 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi064/log 2023-07-14T18:27:45.988 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:27:46.299 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi105/log 2023-07-14T18:27:46.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:27:46.500 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi144/log 2023-07-14T18:27:46.501 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T18:27:46.702 INFO:tasks.cephadm:Removing cluster... 2023-07-14T18:27:46.703 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force 2023-07-14T18:27:46.996 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:47.461 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force 2023-07-14T18:27:47.757 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:48.212 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 5441c8fe-2272-11ee-9b33-001a4aab830c --force 2023-07-14T18:27:48.519 INFO:teuthology.orchestra.run.smithi144.stdout:Deleting cluster with fsid: 5441c8fe-2272-11ee-9b33-001a4aab830c 2023-07-14T18:27:48.970 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T18:27:48.970 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:27:48.992 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:27:49.013 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T18:27:49.031 INFO:tasks.cephadm:Teardown complete 2023-07-14T18:27:49.032 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T18:27:49.043 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T18:27:49.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:27:49.047 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:27:49.057 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T18:27:54.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T18:27:54.206 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T18:27:54.207 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:27:54.210 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:27:54.214 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:27:54.227 INFO:teuthology.orchestra.run.smithi064.stderr:bash: ntpq: command not found 2023-07-14T18:27:54.231 INFO:teuthology.orchestra.run.smithi105.stderr:bash: ntpq: command not found 2023-07-14T18:27:54.233 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:27:54.234 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2023-07-14T18:27:54.234 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 +1213us[+1277us] +/- 79ms 2023-07-14T18:27:54.234 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -3444us[-3380us] +/- 63ms 2023-07-14T18:27:54.234 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 22 +3438us[+3438us] +/- 83ms 2023-07-14T18:27:54.235 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:27:54.235 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:27:54.236 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2023-07-14T18:27:54.236 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +1373us[+1373us] +/- 79ms 2023-07-14T18:27:54.236 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 7 377 89 -3249us[-3221us] +/- 62ms 2023-07-14T18:27:54.236 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +3610us[+3610us] +/- 83ms 2023-07-14T18:27:54.236 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:27:54.237 INFO:teuthology.orchestra.run.smithi144.stderr:bash: ntpq: command not found 2023-07-14T18:27:54.239 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:27:54.239 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2023-07-14T18:27:54.240 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +1229us[+1248us] +/- 79ms 2023-07-14T18:27:54.240 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -3437us[-3419us] +/- 63ms 2023-07-14T18:27:54.240 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +3454us[+3473us] +/- 83ms 2023-07-14T18:27:54.240 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:27:54.241 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T18:27:54.256 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T18:27:54.257 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T18:27:54.269 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:27:54.382 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:27:54.484 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T18:27:54.585 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:27:54.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:54.621 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:27:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:54.655 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:27:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:54.689 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T18:27:54.707 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T18:27:54.728 INFO:teuthology.task.internal:Duration was 1699.399189 seconds 2023-07-14T18:27:54.729 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T18:27:54.744 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T18:27:54.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:27:54.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:27:54.751 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T18:27:54.796 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:27:54.796 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:27:54.800 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:27:55.182 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T18:27:55.183 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2023-07-14T18:27:55.183 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:27:55.208 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2023-07-14T18:27:55.209 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:27:55.232 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2023-07-14T18:27:55.232 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T18:27:55.258 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T18:27:55.259 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:27:55.261 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:27:55.278 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T18:27:55.351 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T18:27:55.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:27:55.354 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:27:55.360 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T18:27:55.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T18:27:55.472 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T18:27:55.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:27:55.500 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:27:55.527 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T18:27:55.554 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T18:27:55.568 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.571 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.573 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.596 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2023-07-14T18:27:55.597 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2023-07-14T18:27:55.620 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2023-07-14T18:27:55.643 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:55.667 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:55.684 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T18:27:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:27:55.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T18:27:55.715 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T18:27:55.716 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi064 2023-07-14T18:27:55.716 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:27:55.755 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi105 2023-07-14T18:27:55.756 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:27:55.791 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010/remote/smithi144 2023-07-14T18:27:55.792 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T18:27:55.828 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T18:27:55.828 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:27:55.831 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:27:55.833 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T18:27:55.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T18:27:55.898 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T18:27:55.898 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T18:27:55.910 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T18:27:55.910 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:27:55.912 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:27:55.914 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T18:27:55.927 INFO:teuthology.orchestra.run.smithi064.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:27 /home/ubuntu/cephtest 2023-07-14T18:27:55.928 INFO:teuthology.orchestra.run.smithi105.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:27 /home/ubuntu/cephtest 2023-07-14T18:27:55.936 INFO:teuthology.orchestra.run.smithi144.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 18:27 /home/ubuntu/cephtest 2023-07-14T18:27:55.937 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T18:27:55.948 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T18:27:55.972 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T18:28:00.999 DEBUG:teuthology.nuke:shortname: smithi064 2023-07-14T18:28:01.000 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:28:10.479 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.174558', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T18:28:10.485 DEBUG:teuthology.nuke:shortname: smithi105 2023-07-14T18:28:10.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:28:10.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.170767', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:28:10.516 DEBUG:teuthology.nuke:shortname: smithi144 2023-07-14T18:28:10.517 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:28:10.532 INFO:teuthology.orchestra.console:Power off smithi064 2023-07-14T18:28:10.533 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:28:10.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:10.589 INFO:teuthology.orchestra.console:Power off smithi105 2023-07-14T18:28:10.590 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:28:10.604 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338010', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 17:51:12.173020', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T18:28:10.619 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:10.676 INFO:teuthology.orchestra.console:Power off smithi144 2023-07-14T18:28:10.677 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T18:28:10.703 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:14.611 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:14.645 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:14.728 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:19.725 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:19.761 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:19.842 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:19.937 INFO:teuthology.orchestra.console:Power off for smithi064 completed 2023-07-14T18:28:24.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:24.954 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi144.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:25.166 INFO:teuthology.orchestra.console:Power off for smithi144 completed 2023-07-14T18:28:29.992 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:35.104 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:40.217 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:45.333 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:50.448 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T18:28:50.660 INFO:teuthology.orchestra.console:Power off for smithi105 completed 2023-07-14T18:28:50.772 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} duration: 1699.39918923378 failure_reason: 'Command failed on smithi064 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup random 0.01''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=59b0efa8feea4d35a002c94bc4564d2a status: fail success: false 2023-07-14T18:28:50.772 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:28:52.237 INFO:teuthology.run:FAIL